2026-02-21T11:26:19.693 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-21T11:26:19.708 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-21T11:26:19.786 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/skanta-2026-02-21_10:47:12-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/62535 branch: wip-bharath21-testing-2026-02-20-1039 description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} email: skanta@redhat.com first_in_suite: false flavor: default job_id: '62535' kernel: branch: distro kdb: 1 sha1: distro last_in_suite: false machine_type: trial name: skanta-2026-02-21_10:47:12-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath21-testing-2026-02-20-1039 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: f7273f3049146bdd7fa7b4a400114872a50228e9 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: f7273f3049146bdd7fa7b4a400114872a50228e9 workunit: branch: wip-bharath21-testing-2026-02-20-1039 sha1: f7273f3049146bdd7fa7b4a400114872a50228e9 owner: scheduled_skanta@soko04.front.sepia.ceph.com priority: 60 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 6481 sha1: f7273f3049146bdd7fa7b4a400114872a50228e9 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath21-testing-2026-02-20-1039 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f7273f3049146bdd7fa7b4a400114872a50228e9/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f7273f3049146bdd7fa7b4a400114872a50228e9 targets: trial189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA7JjawG9yyIWnaNl+38KEfkAxcWbGPfMI8jsHoQsHZqoTq6Ne+slPqypilubfYMwetJGNsvCt+dedZfVut73xA= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-21_10:47:12 tube: trial user: skanta verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.1540699 2026-02-21T11:26:19.787 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f7273f3049146bdd7fa7b4a400114872a50228e9/qa; will attempt to use it 2026-02-21T11:26:19.787 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f7273f3049146bdd7fa7b4a400114872a50228e9/qa/tasks 2026-02-21T11:26:19.787 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-21T11:26:19.789 INFO:teuthology.task.internal:Checking packages... 2026-02-21T11:26:19.789 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f7273f3049146bdd7fa7b4a400114872a50228e9' 2026-02-21T11:26:19.789 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-21T11:26:19.789 INFO:teuthology.packaging:ref: None 2026-02-21T11:26:19.789 INFO:teuthology.packaging:tag: None 2026-02-21T11:26:19.789 INFO:teuthology.packaging:branch: wip-bharath21-testing-2026-02-20-1039 2026-02-21T11:26:19.789 INFO:teuthology.packaging:sha1: f7273f3049146bdd7fa7b4a400114872a50228e9 2026-02-21T11:26:19.789 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath21-testing-2026-02-20-1039 2026-02-21T11:26:19.909 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5397.gf7273f30 2026-02-21T11:26:19.910 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-21T11:26:19.923 INFO:teuthology.task.internal:no buildpackages task found 2026-02-21T11:26:19.923 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-21T11:26:19.940 INFO:teuthology.task.internal:Saving configuration 2026-02-21T11:26:19.948 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-21T11:26:19.962 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-21T11:26:20.058 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial189.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/skanta-2026-02-21_10:47:12-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/62535', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-21 11:22:55.900025', 'locked_by': 'scheduled_skanta@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA7JjawG9yyIWnaNl+38KEfkAxcWbGPfMI8jsHoQsHZqoTq6Ne+slPqypilubfYMwetJGNsvCt+dedZfVut73xA='} 2026-02-21T11:26:20.058 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-21T11:26:20.069 INFO:teuthology.task.internal:roles: ubuntu@trial189.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-21T11:26:20.069 INFO:teuthology.run_tasks:Running task console_log... 2026-02-21T11:26:20.145 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x73e1c9ce6d40>, signals=[15]) 2026-02-21T11:26:20.145 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-21T11:26:20.155 INFO:teuthology.task.internal:Opening connections... 2026-02-21T11:26:20.155 DEBUG:teuthology.task.internal:connecting to ubuntu@trial189.front.sepia.ceph.com 2026-02-21T11:26:20.156 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-21T11:26:20.222 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-21T11:26:20.234 DEBUG:teuthology.orchestra.run.trial189:> uname -m 2026-02-21T11:26:20.293 INFO:teuthology.orchestra.run.trial189.stdout:x86_64 2026-02-21T11:26:20.294 DEBUG:teuthology.orchestra.run.trial189:> cat /etc/os-release 2026-02-21T11:26:20.349 INFO:teuthology.orchestra.run.trial189.stdout:NAME="CentOS Stream" 2026-02-21T11:26:20.349 INFO:teuthology.orchestra.run.trial189.stdout:VERSION="9" 2026-02-21T11:26:20.349 INFO:teuthology.orchestra.run.trial189.stdout:ID="centos" 2026-02-21T11:26:20.350 INFO:teuthology.orchestra.run.trial189.stdout:ID_LIKE="rhel fedora" 2026-02-21T11:26:20.350 INFO:teuthology.orchestra.run.trial189.stdout:VERSION_ID="9" 2026-02-21T11:26:20.350 INFO:teuthology.orchestra.run.trial189.stdout:PLATFORM_ID="platform:el9" 2026-02-21T11:26:20.350 INFO:teuthology.orchestra.run.trial189.stdout:PRETTY_NAME="CentOS Stream 9" 2026-02-21T11:26:20.350 INFO:teuthology.orchestra.run.trial189.stdout:ANSI_COLOR="0;31" 2026-02-21T11:26:20.350 INFO:teuthology.orchestra.run.trial189.stdout:LOGO="fedora-logo-icon" 2026-02-21T11:26:20.350 INFO:teuthology.orchestra.run.trial189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2026-02-21T11:26:20.350 INFO:teuthology.orchestra.run.trial189.stdout:HOME_URL="https://centos.org/" 2026-02-21T11:26:20.350 INFO:teuthology.orchestra.run.trial189.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2026-02-21T11:26:20.350 INFO:teuthology.orchestra.run.trial189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2026-02-21T11:26:20.350 INFO:teuthology.orchestra.run.trial189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2026-02-21T11:26:20.350 INFO:teuthology.lock.ops:Updating trial189.front.sepia.ceph.com on lock server 2026-02-21T11:26:20.412 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-21T11:26:20.424 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-21T11:26:20.436 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-21T11:26:20.437 DEBUG:teuthology.orchestra.run.trial189:> test '!' -e /home/ubuntu/cephtest 2026-02-21T11:26:20.452 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-21T11:26:20.464 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-21T11:26:20.464 DEBUG:teuthology.orchestra.run.trial189:> test -z $(ls -A /var/lib/ceph) 2026-02-21T11:26:20.509 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-21T11:26:20.605 INFO:teuthology.run_tasks:Running task kernel... 2026-02-21T11:26:20.627 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-21T11:26:20.627 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-21T11:26:20.627 DEBUG:teuthology.orchestra.run.trial189:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-21T11:26:20.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T11:26:20.642 DEBUG:teuthology.orchestra.run.trial189:> uname -r 2026-02-21T11:26:20.698 INFO:teuthology.orchestra.run.trial189.stdout:5.14.0-681.el9.x86_64 2026-02-21T11:26:20.698 INFO:teuthology.task.kernel:Running kernel on trial189: 5.14.0-681.el9.x86_64 2026-02-21T11:26:20.698 DEBUG:teuthology.orchestra.run.trial189:> sudo yum install -y kernel 2026-02-21T11:26:21.389 INFO:teuthology.orchestra.run.trial189.stdout:CentOS Stream 9 - BaseOS 29 MB/s | 8.9 MB 00:00 2026-02-21T11:26:24.559 INFO:teuthology.orchestra.run.trial189.stdout:CentOS Stream 9 - AppStream 10 MB/s | 27 MB 00:02 2026-02-21T11:26:35.390 INFO:teuthology.orchestra.run.trial189.stdout:CentOS Stream 9 - CRB 937 kB/s | 7.8 MB 00:08 2026-02-21T11:26:37.105 INFO:teuthology.orchestra.run.trial189.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 20 kB 00:01 2026-02-21T11:26:37.394 INFO:teuthology.orchestra.run.trial189.stdout:Extra Packages for Enterprise Linux 89 MB/s | 20 MB 00:00 2026-02-21T11:26:40.596 INFO:teuthology.orchestra.run.trial189.stdout:lab-extras 2.5 MB/s | 50 kB 00:00 2026-02-21T11:26:41.601 INFO:teuthology.orchestra.run.trial189.stdout:Package kernel-5.14.0-645.el9.x86_64 is already installed. 2026-02-21T11:26:41.602 INFO:teuthology.orchestra.run.trial189.stdout:Package kernel-5.14.0-681.el9.x86_64 is already installed. 2026-02-21T11:26:41.620 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:26:41.621 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:26:41.622 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:26:41.645 DEBUG:teuthology.orchestra.run.trial189:> echo no | sudo yum reinstall kernel || true 2026-02-21T11:26:41.946 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:01 ago on Sat 21 Feb 2026 11:26:40 AM UTC. 2026-02-21T11:26:42.026 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:26:42.027 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:26:42.027 INFO:teuthology.orchestra.run.trial189.stdout: Package Architecture Version Repository Size 2026-02-21T11:26:42.027 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:26:42.027 INFO:teuthology.orchestra.run.trial189.stdout:Reinstalling: 2026-02-21T11:26:42.027 INFO:teuthology.orchestra.run.trial189.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-21T11:26:42.027 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:26:42.027 INFO:teuthology.orchestra.run.trial189.stdout:Transaction Summary 2026-02-21T11:26:42.028 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:26:42.028 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:26:42.028 INFO:teuthology.orchestra.run.trial189.stdout:Total download size: 803 k 2026-02-21T11:26:42.028 INFO:teuthology.orchestra.run.trial189.stdout:Installed size: 0 2026-02-21T11:26:42.028 INFO:teuthology.orchestra.run.trial189.stderr:Operation aborted. 2026-02-21T11:26:42.046 INFO:teuthology.orchestra.run.trial189.stdout:Is this ok [y/N]: 2026-02-21T11:26:42.046 DEBUG:teuthology.orchestra.run.trial189:> sudo yum reinstall -y kernel || true 2026-02-21T11:26:42.346 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:02 ago on Sat 21 Feb 2026 11:26:40 AM UTC. 2026-02-21T11:26:42.426 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:26:42.427 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:26:42.427 INFO:teuthology.orchestra.run.trial189.stdout: Package Architecture Version Repository Size 2026-02-21T11:26:42.427 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:26:42.427 INFO:teuthology.orchestra.run.trial189.stdout:Reinstalling: 2026-02-21T11:26:42.427 INFO:teuthology.orchestra.run.trial189.stdout: kernel x86_64 5.14.0-681.el9 baseos 803 k 2026-02-21T11:26:42.427 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:26:42.427 INFO:teuthology.orchestra.run.trial189.stdout:Transaction Summary 2026-02-21T11:26:42.427 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:26:42.428 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:26:42.428 INFO:teuthology.orchestra.run.trial189.stdout:Total download size: 803 k 2026-02-21T11:26:42.428 INFO:teuthology.orchestra.run.trial189.stdout:Installed size: 0 2026-02-21T11:26:42.428 INFO:teuthology.orchestra.run.trial189.stdout:Downloading Packages: 2026-02-21T11:26:42.513 INFO:teuthology.orchestra.run.trial189.stdout:kernel-5.14.0-681.el9.x86_64.rpm 30 MB/s | 803 kB 00:00 2026-02-21T11:26:42.513 INFO:teuthology.orchestra.run.trial189.stdout:-------------------------------------------------------------------------------- 2026-02-21T11:26:42.513 INFO:teuthology.orchestra.run.trial189.stdout:Total 9.1 MB/s | 803 kB 00:00 2026-02-21T11:26:42.523 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction check 2026-02-21T11:26:42.557 INFO:teuthology.orchestra.run.trial189.stdout:Transaction check succeeded. 2026-02-21T11:26:42.557 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction test 2026-02-21T11:26:42.561 INFO:teuthology.orchestra.run.trial189.stdout:Transaction test succeeded. 2026-02-21T11:26:42.561 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction 2026-02-21T11:26:42.585 INFO:teuthology.orchestra.run.trial189.stdout: Preparing : 1/1 2026-02-21T11:26:42.589 INFO:teuthology.orchestra.run.trial189.stdout: Reinstalling : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-21T11:26:42.589 INFO:teuthology.orchestra.run.trial189.stdout: Cleanup : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-21T11:26:42.700 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: kernel-5.14.0-681.el9.x86_64 2/2 2026-02-21T11:26:42.700 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 1/2 2026-02-21T11:26:42.755 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : kernel-5.14.0-681.el9.x86_64 2/2 2026-02-21T11:26:42.756 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:26:42.756 INFO:teuthology.orchestra.run.trial189.stdout:Reinstalled: 2026-02-21T11:26:42.756 INFO:teuthology.orchestra.run.trial189.stdout: kernel-5.14.0-681.el9.x86_64 2026-02-21T11:26:42.756 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:26:42.756 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:26:42.782 DEBUG:teuthology.orchestra.run.trial189:> rpm -q kernel | sort -rV | head -n 1 2026-02-21T11:26:42.807 INFO:teuthology.orchestra.run.trial189.stdout:kernel-5.14.0-681.el9.x86_64 2026-02-21T11:26:42.808 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-681.el9.x86_64 2026-02-21T11:26:42.808 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-21T11:26:42.809 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-21T11:26:42.809 DEBUG:teuthology.orchestra.run.trial189:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-21T11:26:42.872 INFO:teuthology.orchestra.run.trial189.stdout:ttyS1 2026-02-21T11:26:42.903 DEBUG:teuthology.parallel:result is None 2026-02-21T11:26:42.903 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-21T11:26:42.911 INFO:teuthology.task.internal:Creating test directory... 2026-02-21T11:26:42.911 DEBUG:teuthology.orchestra.run.trial189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-21T11:26:42.958 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-21T11:26:42.970 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-21T11:26:42.984 INFO:teuthology.task.internal:Creating archive directory... 2026-02-21T11:26:42.984 DEBUG:teuthology.orchestra.run.trial189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-21T11:26:43.018 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-21T11:26:43.029 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-21T11:26:43.029 DEBUG:teuthology.orchestra.run.trial189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-21T11:26:43.081 INFO:teuthology.orchestra.run.trial189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:26:43.089 INFO:teuthology.orchestra.run.trial189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:26:43.090 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-21T11:26:43.106 INFO:teuthology.task.internal:Configuring sudo... 2026-02-21T11:26:43.106 DEBUG:teuthology.orchestra.run.trial189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-21T11:26:43.155 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-21T11:26:43.171 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-21T11:26:43.172 DEBUG:teuthology.orchestra.run.trial189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-21T11:26:43.212 DEBUG:teuthology.orchestra.run.trial189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-21T11:26:43.560 DEBUG:teuthology.orchestra.run.trial189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-21T11:26:43.586 DEBUG:teuthology.orchestra.run.trial189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-21T11:26:43.847 DEBUG:teuthology.orchestra.run.trial189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-21T11:26:43.871 DEBUG:teuthology.orchestra.run.trial189:> set -ex 2026-02-21T11:26:43.871 DEBUG:teuthology.orchestra.run.trial189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-21T11:26:43.934 DEBUG:teuthology.orchestra.run.trial189:> sudo service rsyslog restart 2026-02-21T11:26:44.002 INFO:teuthology.orchestra.run.trial189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-21T11:26:44.397 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-21T11:26:44.411 INFO:teuthology.task.internal:Starting timer... 2026-02-21T11:26:44.411 INFO:teuthology.run_tasks:Running task pcp... 2026-02-21T11:26:44.436 INFO:teuthology.run_tasks:Running task selinux... 2026-02-21T11:26:44.457 DEBUG:teuthology.orchestra.run.trial189:> sudo service auditd rotate 2026-02-21T11:26:44.485 INFO:teuthology.orchestra.run.trial189.stdout:Rotating logs: 2026-02-21T11:26:44.487 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-21T11:26:44.487 DEBUG:teuthology.orchestra.run.trial189:> /usr/sbin/getenforce 2026-02-21T11:26:44.543 INFO:teuthology.orchestra.run.trial189.stdout:Permissive 2026-02-21T11:26:44.544 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@trial189.front.sepia.ceph.com': 'permissive'} 2026-02-21T11:26:44.544 DEBUG:teuthology.orchestra.run.trial189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-21T11:26:44.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T11:26:44.609 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-21T11:26:44.609 DEBUG:teuthology.orchestra.run.trial189:> sudo /usr/sbin/setenforce permissive 2026-02-21T11:26:44.673 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-21T11:26:44.689 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-21T11:26:44.700 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-21T11:26:44.702 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-21T11:29:23.726 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial189.front.sepia.ceph.com')] 2026-02-21T11:29:23.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial189.front.sepia.ceph.com' 2026-02-21T11:29:23.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-21T11:29:23.801 DEBUG:teuthology.orchestra.run.trial189:> true 2026-02-21T11:29:23.876 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial189.front.sepia.ceph.com' 2026-02-21T11:29:23.877 INFO:teuthology.run_tasks:Running task clock... 2026-02-21T11:29:23.885 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-21T11:29:23.885 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-21T11:29:23.886 DEBUG:teuthology.orchestra.run.trial189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-21T11:29:23.944 INFO:teuthology.orchestra.run.trial189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2026-02-21T11:29:23.956 INFO:teuthology.orchestra.run.trial189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2026-02-21T11:29:23.977 INFO:teuthology.orchestra.run.trial189.stderr:sudo: ntpd: command not found 2026-02-21T11:29:23.986 INFO:teuthology.orchestra.run.trial189.stdout:506 Cannot talk to daemon 2026-02-21T11:29:23.997 INFO:teuthology.orchestra.run.trial189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2026-02-21T11:29:24.010 INFO:teuthology.orchestra.run.trial189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2026-02-21T11:29:24.056 INFO:teuthology.orchestra.run.trial189.stderr:bash: line 1: ntpq: command not found 2026-02-21T11:29:24.075 INFO:teuthology.orchestra.run.trial189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-21T11:29:24.076 INFO:teuthology.orchestra.run.trial189.stdout:=============================================================================== 2026-02-21T11:29:24.076 INFO:teuthology.orchestra.run.trial189.stdout:^? ntp1.wiktel.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-21T11:29:24.076 INFO:teuthology.orchestra.run.trial189.stdout:^? 144.202.66.214.vultruser> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-21T11:29:24.076 INFO:teuthology.orchestra.run.trial189.stdout:^? pool-138-89-14-60.mad.ea> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-21T11:29:24.076 INFO:teuthology.orchestra.run.trial189.stdout:^? 167-248-62-201.oa02.lnk0> 0 6 0 - +0ns[ +0ns] +/- 0ns 2026-02-21T11:29:24.076 INFO:teuthology.run_tasks:Running task install... 2026-02-21T11:29:24.092 DEBUG:teuthology.task.install:project ceph 2026-02-21T11:29:24.092 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f7273f3049146bdd7fa7b4a400114872a50228e9'}} 2026-02-21T11:29:24.092 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'f7273f3049146bdd7fa7b4a400114872a50228e9'} 2026-02-21T11:29:24.093 INFO:teuthology.task.install:Using flavor: default 2026-02-21T11:29:24.103 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-21T11:29:24.103 INFO:teuthology.task.install:extra packages: [] 2026-02-21T11:29:24.103 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f7273f3049146bdd7fa7b4a400114872a50228e9', 'tag': None, 'wait_for_package': False} 2026-02-21T11:29:24.105 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f7273f3049146bdd7fa7b4a400114872a50228e9 2026-02-21T11:29:24.271 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-bharath21-testing-2026-02-20-1039/f7273f3049146bdd7fa7b4a400114872a50228e9/centos/9/flavors/default/ 2026-02-21T11:29:24.271 INFO:teuthology.task.install.rpm:Package version is 20.3.0-5397.gf7273f30 2026-02-21T11:29:24.398 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath21-testing-2026-02-20-1039/f7273f3049146bdd7fa7b4a400114872a50228e9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath21-testing-2026-02-20-1039/f7273f3049146bdd7fa7b4a400114872a50228e9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath21-testing-2026-02-20-1039/f7273f3049146bdd7fa7b4a400114872a50228e9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2026-02-21T11:29:24.398 DEBUG:teuthology.orchestra.run.trial189:> set -ex 2026-02-21T11:29:24.398 DEBUG:teuthology.orchestra.run.trial189:> sudo dd of=/etc/yum.repos.d/ceph.repo 2026-02-21T11:29:24.425 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-osd, ceph-osd-classic, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2026-02-21T11:29:24.425 DEBUG:teuthology.orchestra.run.trial189:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f7273f3049146bdd7fa7b4a400114872a50228e9/;g' /etc/yum.repos.d/ceph.repo ; fi 2026-02-21T11:29:24.489 DEBUG:teuthology.orchestra.run.trial189:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2026-02-21T11:29:24.562 DEBUG:teuthology.orchestra.run.trial189:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2026-02-21T11:29:24.626 INFO:teuthology.orchestra.run.trial189.stdout:check_obsoletes = 1 2026-02-21T11:29:24.627 DEBUG:teuthology.orchestra.run.trial189:> sudo yum clean all 2026-02-21T11:29:24.784 INFO:teuthology.orchestra.run.trial189.stdout:41 files removed 2026-02-21T11:29:24.800 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install ceph-radosgw 2026-02-21T11:29:25.072 INFO:teuthology.orchestra.run.trial189.stdout:ceph packages for x86_64 723 kB/s | 94 kB 00:00 2026-02-21T11:29:25.218 INFO:teuthology.orchestra.run.trial189.stdout:ceph noarch packages 207 kB/s | 25 kB 00:00 2026-02-21T11:29:25.346 INFO:teuthology.orchestra.run.trial189.stdout:ceph source packages 19 kB/s | 2.1 kB 00:00 2026-02-21T11:29:26.105 INFO:teuthology.orchestra.run.trial189.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.9 MB 00:00 2026-02-21T11:29:29.894 INFO:teuthology.orchestra.run.trial189.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 27 MB 00:03 2026-02-21T11:29:33.858 INFO:teuthology.orchestra.run.trial189.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 7.8 MB 00:01 2026-02-21T11:29:34.759 INFO:teuthology.orchestra.run.trial189.stdout:CentOS Stream 9 - Extras packages 116 kB/s | 20 kB 00:00 2026-02-21T11:29:36.403 INFO:teuthology.orchestra.run.trial189.stdout:Extra Packages for Enterprise Linux 13 MB/s | 20 MB 00:01 2026-02-21T11:29:39.635 INFO:teuthology.orchestra.run.trial189.stdout:lab-extras 1.4 MB/s | 50 kB 00:00 2026-02-21T11:29:40.821 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:29:40.822 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:29:40.822 INFO:teuthology.orchestra.run.trial189.stdout: Package Arch Version Repository Size 2026-02-21T11:29:40.822 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:29:40.822 INFO:teuthology.orchestra.run.trial189.stdout:Installing: 2026-02-21T11:29:40.822 INFO:teuthology.orchestra.run.trial189.stdout: ceph-radosgw x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 13 M 2026-02-21T11:29:40.822 INFO:teuthology.orchestra.run.trial189.stdout:Upgrading: 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: librados2 x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 3.5 M 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: librbd1 x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 2.8 M 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout:Installing dependencies: 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: boost-program-options x86_64 1.75.0-13.el9 appstream 104 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: ceph-base x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 5.2 M 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: ceph-common x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 24 M 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: ceph-selinux x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 25 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: libarrow x86_64 9.0.0-15.el9 epel 4.4 M 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: libarrow-doc noarch 9.0.0-15.el9 epel 25 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: libcephfs2 x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 908 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: libnbd x86_64 1.20.3-4.el9 appstream 164 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: liboath x86_64 2.6.12-1.el9 epel 49 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: libradosstriper1 x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 248 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: librgw2 x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 6.5 M 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: mailcap noarch 2.1.49-5.el9 baseos 33 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: parquet-libs x86_64 9.0.0-15.el9 epel 838 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: pciutils x86_64 3.7.0-7.el9 baseos 93 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: python3-ceph-argparse x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 45 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: python3-ceph-common x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 185 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: python3-cephfs x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 203 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: python3-rados x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 324 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: python3-rbd x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 305 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: python3-rgw x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 100 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: qatlib x86_64 25.08.0-2.el9 appstream 240 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: qatzip-libs x86_64 1.3.1-1.el9 appstream 66 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: thrift x86_64 0.15.0-4.el9 epel 1.6 M 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout:Installing weak dependencies: 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: libcephfs-daemon x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 37 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: libcephfs-proxy2 x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 25 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: nvme-cli x86_64 2.16-1.el9 baseos 1.2 M 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: qatlib-service x86_64 25.08.0-2.el9 appstream 37 k 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout:Transaction Summary 2026-02-21T11:29:40.823 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:29:40.824 INFO:teuthology.orchestra.run.trial189.stdout:Install 35 Packages 2026-02-21T11:29:40.824 INFO:teuthology.orchestra.run.trial189.stdout:Upgrade 2 Packages 2026-02-21T11:29:40.824 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:29:40.824 INFO:teuthology.orchestra.run.trial189.stdout:Total download size: 68 M 2026-02-21T11:29:40.824 INFO:teuthology.orchestra.run.trial189.stdout:Downloading Packages: 2026-02-21T11:29:41.939 INFO:teuthology.orchestra.run.trial189.stdout:(1/37): ceph-base-20.3.0-5397.gf7273f30.el9.x86 5.9 MB/s | 5.2 MB 00:00 2026-02-21T11:29:41.959 INFO:teuthology.orchestra.run.trial189.stdout:(2/37): ceph-selinux-20.3.0-5397.gf7273f30.el9. 1.2 MB/s | 25 kB 00:00 2026-02-21T11:29:41.980 INFO:teuthology.orchestra.run.trial189.stdout:(3/37): libcephfs-daemon-20.3.0-5397.gf7273f30. 1.8 MB/s | 37 kB 00:00 2026-02-21T11:29:42.002 INFO:teuthology.orchestra.run.trial189.stdout:(4/37): libcephfs-proxy2-20.3.0-5397.gf7273f30. 1.1 MB/s | 25 kB 00:00 2026-02-21T11:29:42.084 INFO:teuthology.orchestra.run.trial189.stdout:(5/37): libcephfs2-20.3.0-5397.gf7273f30.el9.x8 11 MB/s | 908 kB 00:00 2026-02-21T11:29:42.113 INFO:teuthology.orchestra.run.trial189.stdout:(6/37): libradosstriper1-20.3.0-5397.gf7273f30. 8.3 MB/s | 248 kB 00:00 2026-02-21T11:29:42.290 INFO:teuthology.orchestra.run.trial189.stdout:(7/37): ceph-radosgw-20.3.0-5397.gf7273f30.el9. 11 MB/s | 13 MB 00:01 2026-02-21T11:29:42.315 INFO:teuthology.orchestra.run.trial189.stdout:(8/37): python3-ceph-argparse-20.3.0-5397.gf727 1.7 MB/s | 45 kB 00:00 2026-02-21T11:29:42.362 INFO:teuthology.orchestra.run.trial189.stdout:(9/37): ceph-common-20.3.0-5397.gf7273f30.el9.x 18 MB/s | 24 MB 00:01 2026-02-21T11:29:42.363 INFO:teuthology.orchestra.run.trial189.stdout:(10/37): python3-ceph-common-20.3.0-5397.gf7273 3.9 MB/s | 185 kB 00:00 2026-02-21T11:29:42.374 INFO:teuthology.orchestra.run.trial189.stdout:(11/37): python3-cephfs-20.3.0-5397.gf7273f30.e 16 MB/s | 203 kB 00:00 2026-02-21T11:29:42.383 INFO:teuthology.orchestra.run.trial189.stdout:(12/37): python3-rados-20.3.0-5397.gf7273f30.el 16 MB/s | 324 kB 00:00 2026-02-21T11:29:42.396 INFO:teuthology.orchestra.run.trial189.stdout:(13/37): python3-rbd-20.3.0-5397.gf7273f30.el9. 13 MB/s | 305 kB 00:00 2026-02-21T11:29:42.406 INFO:teuthology.orchestra.run.trial189.stdout:(14/37): python3-rgw-20.3.0-5397.gf7273f30.el9. 4.2 MB/s | 100 kB 00:00 2026-02-21T11:29:42.407 INFO:teuthology.orchestra.run.trial189.stdout:(15/37): mailcap-2.1.49-5.el9.noarch.rpm 3.1 MB/s | 33 kB 00:00 2026-02-21T11:29:42.415 INFO:teuthology.orchestra.run.trial189.stdout:(16/37): pciutils-3.7.0-7.el9.x86_64.rpm 11 MB/s | 93 kB 00:00 2026-02-21T11:29:42.432 INFO:teuthology.orchestra.run.trial189.stdout:(17/37): librgw2-20.3.0-5397.gf7273f30.el9.x86_ 20 MB/s | 6.5 MB 00:00 2026-02-21T11:29:42.436 INFO:teuthology.orchestra.run.trial189.stdout:(18/37): nvme-cli-2.16-1.el9.x86_64.rpm 38 MB/s | 1.2 MB 00:00 2026-02-21T11:29:43.091 INFO:teuthology.orchestra.run.trial189.stdout:(19/37): boost-program-options-1.75.0-13.el9.x8 154 kB/s | 104 kB 00:00 2026-02-21T11:29:43.162 INFO:teuthology.orchestra.run.trial189.stdout:(20/37): libnbd-1.20.3-4.el9.x86_64.rpm 224 kB/s | 164 kB 00:00 2026-02-21T11:29:43.169 INFO:teuthology.orchestra.run.trial189.stdout:(21/37): libpmemobj-1.12.1-1.el9.x86_64.rpm 219 kB/s | 160 kB 00:00 2026-02-21T11:29:43.173 INFO:teuthology.orchestra.run.trial189.stdout:(22/37): librabbitmq-0.11.0-7.el9.x86_64.rpm 549 kB/s | 45 kB 00:00 2026-02-21T11:29:43.489 INFO:teuthology.orchestra.run.trial189.stdout:(23/37): qatlib-25.08.0-2.el9.x86_64.rpm 759 kB/s | 240 kB 00:00 2026-02-21T11:29:43.566 INFO:teuthology.orchestra.run.trial189.stdout:(24/37): lttng-ust-2.12.0-6.el9.x86_64.rpm 737 kB/s | 292 kB 00:00 2026-02-21T11:29:43.571 INFO:teuthology.orchestra.run.trial189.stdout:(25/37): qatlib-service-25.08.0-2.el9.x86_64.rp 449 kB/s | 37 kB 00:00 2026-02-21T11:29:43.649 INFO:teuthology.orchestra.run.trial189.stdout:(26/37): qatzip-libs-1.3.1-1.el9.x86_64.rpm 797 kB/s | 66 kB 00:00 2026-02-21T11:29:43.724 INFO:teuthology.orchestra.run.trial189.stdout:(27/37): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.0 MB/s | 308 kB 00:00 2026-02-21T11:29:43.744 INFO:teuthology.orchestra.run.trial189.stdout:(28/37): libarrow-doc-9.0.0-15.el9.noarch.rpm 1.3 MB/s | 25 kB 00:00 2026-02-21T11:29:43.764 INFO:teuthology.orchestra.run.trial189.stdout:(29/37): liboath-2.6.12-1.el9.x86_64.rpm 2.4 MB/s | 49 kB 00:00 2026-02-21T11:29:43.785 INFO:teuthology.orchestra.run.trial189.stdout:(30/37): libunwind-1.6.2-1.el9.x86_64.rpm 3.1 MB/s | 67 kB 00:00 2026-02-21T11:29:43.839 INFO:teuthology.orchestra.run.trial189.stdout:(31/37): parquet-libs-9.0.0-15.el9.x86_64.rpm 15 MB/s | 838 kB 00:00 2026-02-21T11:29:43.860 INFO:teuthology.orchestra.run.trial189.stdout:(32/37): re2-20211101-20.el9.x86_64.rpm 8.9 MB/s | 191 kB 00:00 2026-02-21T11:29:43.871 INFO:teuthology.orchestra.run.trial189.stdout:(33/37): libarrow-9.0.0-15.el9.x86_64.rpm 20 MB/s | 4.4 MB 00:00 2026-02-21T11:29:43.904 INFO:teuthology.orchestra.run.trial189.stdout:(34/37): thrift-0.15.0-4.el9.x86_64.rpm 36 MB/s | 1.6 MB 00:00 2026-02-21T11:29:43.944 INFO:teuthology.orchestra.run.trial189.stdout:(35/37): librdkafka-1.6.1-102.el9.x86_64.rpm 846 kB/s | 662 kB 00:00 2026-02-21T11:29:44.031 INFO:teuthology.orchestra.run.trial189.stdout:(36/37): librbd1-20.3.0-5397.gf7273f30.el9.x86_ 22 MB/s | 2.8 MB 00:00 2026-02-21T11:29:44.035 INFO:teuthology.orchestra.run.trial189.stdout:(37/37): librados2-20.3.0-5397.gf7273f30.el9.x8 21 MB/s | 3.5 MB 00:00 2026-02-21T11:29:44.038 INFO:teuthology.orchestra.run.trial189.stdout:-------------------------------------------------------------------------------- 2026-02-21T11:29:44.038 INFO:teuthology.orchestra.run.trial189.stdout:Total 21 MB/s | 68 MB 00:03 2026-02-21T11:29:44.146 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction check 2026-02-21T11:29:44.167 INFO:teuthology.orchestra.run.trial189.stdout:Transaction check succeeded. 2026-02-21T11:29:44.167 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction test 2026-02-21T11:29:44.439 INFO:teuthology.orchestra.run.trial189.stdout:Transaction test succeeded. 2026-02-21T11:29:44.439 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction 2026-02-21T11:29:44.749 INFO:teuthology.orchestra.run.trial189.stdout: Preparing : 1/1 2026-02-21T11:29:44.901 INFO:teuthology.orchestra.run.trial189.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/39 2026-02-21T11:29:44.906 INFO:teuthology.orchestra.run.trial189.stdout: Upgrading : librados2-2:20.3.0-5397.gf7273f30.el9.x86_64 2/39 2026-02-21T11:29:44.959 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: librados2-2:20.3.0-5397.gf7273f30.el9.x86_64 2/39 2026-02-21T11:29:44.960 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libcephfs2-2:20.3.0-5397.gf7273f30.el9.x86_64 3/39 2026-02-21T11:29:44.974 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: libcephfs2-2:20.3.0-5397.gf7273f30.el9.x86_64 3/39 2026-02-21T11:29:44.977 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libcephfs-daemon-2:20.3.0-5397.gf7273f30.el9.x86_6 4/39 2026-02-21T11:29:44.977 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libcephfs-proxy2-2:20.3.0-5397.gf7273f30.el9.x86_6 5/39 2026-02-21T11:29:45.005 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: libcephfs-proxy2-2:20.3.0-5397.gf7273f30.el9.x86_6 5/39 2026-02-21T11:29:45.007 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-rados-2:20.3.0-5397.gf7273f30.el9.x86_64 6/39 2026-02-21T11:29:45.017 INFO:teuthology.orchestra.run.trial189.stdout: Installing : liboath-2.6.12-1.el9.x86_64 7/39 2026-02-21T11:29:45.019 INFO:teuthology.orchestra.run.trial189.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 8/39 2026-02-21T11:29:45.022 INFO:teuthology.orchestra.run.trial189.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 9/39 2026-02-21T11:29:45.026 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 10/39 2026-02-21T11:29:45.037 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-ceph-argparse-2:20.3.0-5397.gf7273f30.el9. 11/39 2026-02-21T11:29:45.049 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-cephfs-2:20.3.0-5397.gf7273f30.el9.x86_64 12/39 2026-02-21T11:29:45.050 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libradosstriper1-2:20.3.0-5397.gf7273f30.el9.x86_6 13/39 2026-02-21T11:29:45.072 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: libradosstriper1-2:20.3.0-5397.gf7273f30.el9.x86_6 13/39 2026-02-21T11:29:45.076 INFO:teuthology.orchestra.run.trial189.stdout: Installing : thrift-0.15.0-4.el9.x86_64 14/39 2026-02-21T11:29:45.107 INFO:teuthology.orchestra.run.trial189.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/39 2026-02-21T11:29:45.110 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libarrow-9.0.0-15.el9.x86_64 16/39 2026-02-21T11:29:45.117 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/39 2026-02-21T11:29:45.119 INFO:teuthology.orchestra.run.trial189.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/39 2026-02-21T11:29:45.124 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libarrow-doc-9.0.0-15.el9.noarch 19/39 2026-02-21T11:29:45.246 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-21T11:29:45.247 INFO:teuthology.orchestra.run.trial189.stdout: Upgrading : librbd1-2:20.3.0-5397.gf7273f30.el9.x86_64 21/39 2026-02-21T11:29:45.274 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: librbd1-2:20.3.0-5397.gf7273f30.el9.x86_64 21/39 2026-02-21T11:29:45.277 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-rbd-2:20.3.0-5397.gf7273f30.el9.x86_64 22/39 2026-02-21T11:29:45.283 INFO:teuthology.orchestra.run.trial189.stdout: Installing : boost-program-options-1.75.0-13.el9.x86_64 23/39 2026-02-21T11:29:45.567 INFO:teuthology.orchestra.run.trial189.stdout: Installing : parquet-libs-9.0.0-15.el9.x86_64 24/39 2026-02-21T11:29:45.568 INFO:teuthology.orchestra.run.trial189.stdout: Installing : librgw2-2:20.3.0-5397.gf7273f30.el9.x86_64 25/39 2026-02-21T11:29:45.585 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: librgw2-2:20.3.0-5397.gf7273f30.el9.x86_64 25/39 2026-02-21T11:29:45.590 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-rgw-2:20.3.0-5397.gf7273f30.el9.x86_64 26/39 2026-02-21T11:29:45.595 INFO:teuthology.orchestra.run.trial189.stdout: Installing : pciutils-3.7.0-7.el9.x86_64 27/39 2026-02-21T11:29:45.597 INFO:teuthology.orchestra.run.trial189.stdout: Installing : qatlib-25.08.0-2.el9.x86_64 28/39 2026-02-21T11:29:45.598 INFO:teuthology.orchestra.run.trial189.stdout: Installing : qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-21T11:29:45.617 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: qatlib-service-25.08.0-2.el9.x86_64 29/39 2026-02-21T11:29:45.733 INFO:teuthology.orchestra.run.trial189.stdout: Installing : qatzip-libs-1.3.1-1.el9.x86_64 30/39 2026-02-21T11:29:45.736 INFO:teuthology.orchestra.run.trial189.stdout: Installing : nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-21T11:29:46.015 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: nvme-cli-2.16-1.el9.x86_64 31/39 2026-02-21T11:29:46.015 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /usr/lib/systemd/system/nvmefc-boot-connections.service. 2026-02-21T11:29:46.016 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:29:46.059 INFO:teuthology.orchestra.run.trial189.stdout: Installing : mailcap-2.1.49-5.el9.noarch 32/39 2026-02-21T11:29:46.061 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-ceph-common-2:20.3.0-5397.gf7273f30.el9.x8 33/39 2026-02-21T11:29:47.159 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-common-2:20.3.0-5397.gf7273f30.el9.x86_64 34/39 2026-02-21T11:29:47.162 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-common-2:20.3.0-5397.gf7273f30.el9.x86_64 34/39 2026-02-21T11:29:47.425 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-common-2:20.3.0-5397.gf7273f30.el9.x86_64 34/39 2026-02-21T11:29:47.429 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-base-2:20.3.0-5397.gf7273f30.el9.x86_64 35/39 2026-02-21T11:29:47.467 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-base-2:20.3.0-5397.gf7273f30.el9.x86_64 35/39 2026-02-21T11:29:47.467 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2026-02-21T11:29:47.467 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2026-02-21T11:29:47.467 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:29:47.470 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-selinux-2:20.3.0-5397.gf7273f30.el9.x86_64 36/39 2026-02-21T11:29:52.040 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-selinux-2:20.3.0-5397.gf7273f30.el9.x86_64 36/39 2026-02-21T11:29:52.040 INFO:teuthology.orchestra.run.trial189.stdout:skipping the directory /sys 2026-02-21T11:29:52.040 INFO:teuthology.orchestra.run.trial189.stdout:skipping the directory /proc 2026-02-21T11:29:52.040 INFO:teuthology.orchestra.run.trial189.stdout:skipping the directory /mnt 2026-02-21T11:29:52.040 INFO:teuthology.orchestra.run.trial189.stdout:skipping the directory /var/tmp 2026-02-21T11:29:52.040 INFO:teuthology.orchestra.run.trial189.stdout:skipping the directory /home 2026-02-21T11:29:52.040 INFO:teuthology.orchestra.run.trial189.stdout:skipping the directory /root 2026-02-21T11:29:52.040 INFO:teuthology.orchestra.run.trial189.stdout:skipping the directory /tmp 2026-02-21T11:29:52.040 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:29:52.618 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-radosgw-2:20.3.0-5397.gf7273f30.el9.x86_64 37/39 2026-02-21T11:29:52.643 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-radosgw-2:20.3.0-5397.gf7273f30.el9.x86_64 37/39 2026-02-21T11:29:52.643 INFO:teuthology.orchestra.run.trial189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T11:29:52.643 INFO:teuthology.orchestra.run.trial189.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2026-02-21T11:29:52.643 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-21T11:29:52.643 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2026-02-21T11:29:52.643 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:29:52.644 INFO:teuthology.orchestra.run.trial189.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-21T11:29:52.657 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 38/39 2026-02-21T11:29:52.657 INFO:teuthology.orchestra.run.trial189.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-21T11:29:52.715 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 39/39 2026-02-21T11:29:52.715 INFO:teuthology.orchestra.run.trial189.stdout:Creating group 'qat' with GID 992. 2026-02-21T11:29:52.716 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:29:53.018 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-base-2:20.3.0-5397.gf7273f30.el9.x86_64 1/39 2026-02-21T11:29:53.018 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-common-2:20.3.0-5397.gf7273f30.el9.x86_64 2/39 2026-02-21T11:29:53.018 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-radosgw-2:20.3.0-5397.gf7273f30.el9.x86_64 3/39 2026-02-21T11:29:53.018 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-selinux-2:20.3.0-5397.gf7273f30.el9.x86_64 4/39 2026-02-21T11:29:53.018 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libcephfs-daemon-2:20.3.0-5397.gf7273f30.el9.x86_6 5/39 2026-02-21T11:29:53.018 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libcephfs-proxy2-2:20.3.0-5397.gf7273f30.el9.x86_6 6/39 2026-02-21T11:29:53.019 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libcephfs2-2:20.3.0-5397.gf7273f30.el9.x86_64 7/39 2026-02-21T11:29:53.019 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libradosstriper1-2:20.3.0-5397.gf7273f30.el9.x86_6 8/39 2026-02-21T11:29:53.019 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : librgw2-2:20.3.0-5397.gf7273f30.el9.x86_64 9/39 2026-02-21T11:29:53.019 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-ceph-argparse-2:20.3.0-5397.gf7273f30.el9. 10/39 2026-02-21T11:29:53.019 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-ceph-common-2:20.3.0-5397.gf7273f30.el9.x8 11/39 2026-02-21T11:29:53.019 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-cephfs-2:20.3.0-5397.gf7273f30.el9.x86_64 12/39 2026-02-21T11:29:53.019 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-rados-2:20.3.0-5397.gf7273f30.el9.x86_64 13/39 2026-02-21T11:29:53.019 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-rbd-2:20.3.0-5397.gf7273f30.el9.x86_64 14/39 2026-02-21T11:29:53.019 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-rgw-2:20.3.0-5397.gf7273f30.el9.x86_64 15/39 2026-02-21T11:29:53.019 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : mailcap-2.1.49-5.el9.noarch 16/39 2026-02-21T11:29:53.019 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : nvme-cli-2.16-1.el9.x86_64 17/39 2026-02-21T11:29:53.019 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : pciutils-3.7.0-7.el9.x86_64 18/39 2026-02-21T11:29:53.020 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : boost-program-options-1.75.0-13.el9.x86_64 19/39 2026-02-21T11:29:53.020 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libnbd-1.20.3-4.el9.x86_64 20/39 2026-02-21T11:29:53.020 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 21/39 2026-02-21T11:29:53.020 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 22/39 2026-02-21T11:29:53.020 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 23/39 2026-02-21T11:29:53.020 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 24/39 2026-02-21T11:29:53.020 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : qatlib-25.08.0-2.el9.x86_64 25/39 2026-02-21T11:29:53.020 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : qatlib-service-25.08.0-2.el9.x86_64 26/39 2026-02-21T11:29:53.020 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : qatzip-libs-1.3.1-1.el9.x86_64 27/39 2026-02-21T11:29:53.020 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 28/39 2026-02-21T11:29:53.020 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libarrow-9.0.0-15.el9.x86_64 29/39 2026-02-21T11:29:53.020 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libarrow-doc-9.0.0-15.el9.noarch 30/39 2026-02-21T11:29:53.020 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : liboath-2.6.12-1.el9.x86_64 31/39 2026-02-21T11:29:53.021 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 32/39 2026-02-21T11:29:53.021 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : parquet-libs-9.0.0-15.el9.x86_64 33/39 2026-02-21T11:29:53.021 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : re2-1:20211101-20.el9.x86_64 34/39 2026-02-21T11:29:53.021 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : thrift-0.15.0-4.el9.x86_64 35/39 2026-02-21T11:29:53.021 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : librados2-2:20.3.0-5397.gf7273f30.el9.x86_64 36/39 2026-02-21T11:29:53.021 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 37/39 2026-02-21T11:29:53.021 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : librbd1-2:20.3.0-5397.gf7273f30.el9.x86_64 38/39 2026-02-21T11:29:53.079 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 39/39 2026-02-21T11:29:53.079 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:29:53.079 INFO:teuthology.orchestra.run.trial189.stdout:Upgraded: 2026-02-21T11:29:53.079 INFO:teuthology.orchestra.run.trial189.stdout: librados2-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.079 INFO:teuthology.orchestra.run.trial189.stdout: librbd1-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.080 INFO:teuthology.orchestra.run.trial189.stdout:Installed: 2026-02-21T11:29:53.080 INFO:teuthology.orchestra.run.trial189.stdout: boost-program-options-1.75.0-13.el9.x86_64 2026-02-21T11:29:53.080 INFO:teuthology.orchestra.run.trial189.stdout: ceph-base-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.080 INFO:teuthology.orchestra.run.trial189.stdout: ceph-common-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.080 INFO:teuthology.orchestra.run.trial189.stdout: ceph-radosgw-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.080 INFO:teuthology.orchestra.run.trial189.stdout: ceph-selinux-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.080 INFO:teuthology.orchestra.run.trial189.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2026-02-21T11:29:53.080 INFO:teuthology.orchestra.run.trial189.stdout: libarrow-9.0.0-15.el9.x86_64 2026-02-21T11:29:53.080 INFO:teuthology.orchestra.run.trial189.stdout: libarrow-doc-9.0.0-15.el9.noarch 2026-02-21T11:29:53.080 INFO:teuthology.orchestra.run.trial189.stdout: libcephfs-daemon-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.080 INFO:teuthology.orchestra.run.trial189.stdout: libcephfs-proxy2-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.080 INFO:teuthology.orchestra.run.trial189.stdout: libcephfs2-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.081 INFO:teuthology.orchestra.run.trial189.stdout: libnbd-1.20.3-4.el9.x86_64 2026-02-21T11:29:53.081 INFO:teuthology.orchestra.run.trial189.stdout: liboath-2.6.12-1.el9.x86_64 2026-02-21T11:29:53.081 INFO:teuthology.orchestra.run.trial189.stdout: libpmemobj-1.12.1-1.el9.x86_64 2026-02-21T11:29:53.081 INFO:teuthology.orchestra.run.trial189.stdout: librabbitmq-0.11.0-7.el9.x86_64 2026-02-21T11:29:53.081 INFO:teuthology.orchestra.run.trial189.stdout: libradosstriper1-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.081 INFO:teuthology.orchestra.run.trial189.stdout: librdkafka-1.6.1-102.el9.x86_64 2026-02-21T11:29:53.081 INFO:teuthology.orchestra.run.trial189.stdout: librgw2-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.081 INFO:teuthology.orchestra.run.trial189.stdout: libunwind-1.6.2-1.el9.x86_64 2026-02-21T11:29:53.081 INFO:teuthology.orchestra.run.trial189.stdout: lttng-ust-2.12.0-6.el9.x86_64 2026-02-21T11:29:53.081 INFO:teuthology.orchestra.run.trial189.stdout: mailcap-2.1.49-5.el9.noarch 2026-02-21T11:29:53.081 INFO:teuthology.orchestra.run.trial189.stdout: nvme-cli-2.16-1.el9.x86_64 2026-02-21T11:29:53.081 INFO:teuthology.orchestra.run.trial189.stdout: parquet-libs-9.0.0-15.el9.x86_64 2026-02-21T11:29:53.082 INFO:teuthology.orchestra.run.trial189.stdout: pciutils-3.7.0-7.el9.x86_64 2026-02-21T11:29:53.082 INFO:teuthology.orchestra.run.trial189.stdout: python3-ceph-argparse-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.082 INFO:teuthology.orchestra.run.trial189.stdout: python3-ceph-common-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.082 INFO:teuthology.orchestra.run.trial189.stdout: python3-cephfs-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.082 INFO:teuthology.orchestra.run.trial189.stdout: python3-rados-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.082 INFO:teuthology.orchestra.run.trial189.stdout: python3-rbd-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.082 INFO:teuthology.orchestra.run.trial189.stdout: python3-rgw-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:29:53.082 INFO:teuthology.orchestra.run.trial189.stdout: qatlib-25.08.0-2.el9.x86_64 2026-02-21T11:29:53.082 INFO:teuthology.orchestra.run.trial189.stdout: qatlib-service-25.08.0-2.el9.x86_64 2026-02-21T11:29:53.082 INFO:teuthology.orchestra.run.trial189.stdout: qatzip-libs-1.3.1-1.el9.x86_64 2026-02-21T11:29:53.082 INFO:teuthology.orchestra.run.trial189.stdout: re2-1:20211101-20.el9.x86_64 2026-02-21T11:29:53.082 INFO:teuthology.orchestra.run.trial189.stdout: thrift-0.15.0-4.el9.x86_64 2026-02-21T11:29:53.083 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:29:53.083 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:29:53.124 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install ceph-test 2026-02-21T11:29:53.439 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:14 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:29:53.523 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:29:53.523 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:29:53.523 INFO:teuthology.orchestra.run.trial189.stdout: Package Arch Version Repository Size 2026-02-21T11:29:53.523 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:29:53.523 INFO:teuthology.orchestra.run.trial189.stdout:Installing: 2026-02-21T11:29:53.523 INFO:teuthology.orchestra.run.trial189.stdout: ceph-test x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 76 M 2026-02-21T11:29:53.523 INFO:teuthology.orchestra.run.trial189.stdout:Installing dependencies: 2026-02-21T11:29:53.523 INFO:teuthology.orchestra.run.trial189.stdout: hwloc-libs x86_64 2.4.1-6.el9 baseos 2.1 M 2026-02-21T11:29:53.523 INFO:teuthology.orchestra.run.trial189.stdout: libatomic x86_64 11.5.0-14.el9 baseos 23 k 2026-02-21T11:29:53.524 INFO:teuthology.orchestra.run.trial189.stdout: libcephsqlite x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 159 k 2026-02-21T11:29:53.524 INFO:teuthology.orchestra.run.trial189.stdout: libxslt x86_64 1.1.34-12.el9 appstream 233 k 2026-02-21T11:29:53.524 INFO:teuthology.orchestra.run.trial189.stdout: lksctp-tools x86_64 1.0.19-2.el9 baseos 94 k 2026-02-21T11:29:53.524 INFO:teuthology.orchestra.run.trial189.stdout: protobuf x86_64 3.14.0-17.el9 appstream 1.0 M 2026-02-21T11:29:53.524 INFO:teuthology.orchestra.run.trial189.stdout: socat x86_64 1.7.4.1-8.el9 appstream 303 k 2026-02-21T11:29:53.524 INFO:teuthology.orchestra.run.trial189.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2026-02-21T11:29:53.524 INFO:teuthology.orchestra.run.trial189.stdout: yaml-cpp x86_64 0.6.3-5.el9 epel 122 k 2026-02-21T11:29:53.524 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:29:53.524 INFO:teuthology.orchestra.run.trial189.stdout:Transaction Summary 2026-02-21T11:29:53.524 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:29:53.524 INFO:teuthology.orchestra.run.trial189.stdout:Install 10 Packages 2026-02-21T11:29:53.524 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:29:53.524 INFO:teuthology.orchestra.run.trial189.stdout:Total download size: 80 M 2026-02-21T11:29:53.524 INFO:teuthology.orchestra.run.trial189.stdout:Installed size: 331 M 2026-02-21T11:29:53.524 INFO:teuthology.orchestra.run.trial189.stdout:Downloading Packages: 2026-02-21T11:29:54.146 INFO:teuthology.orchestra.run.trial189.stdout:(1/10): libcephsqlite-20.3.0-5397.gf7273f30.el9 1.6 MB/s | 159 kB 00:00 2026-02-21T11:29:54.700 INFO:teuthology.orchestra.run.trial189.stdout:(2/10): libatomic-11.5.0-14.el9.x86_64.rpm 41 kB/s | 23 kB 00:00 2026-02-21T11:29:55.014 INFO:teuthology.orchestra.run.trial189.stdout:(3/10): lksctp-tools-1.0.19-2.el9.x86_64.rpm 299 kB/s | 94 kB 00:00 2026-02-21T11:29:55.172 INFO:teuthology.orchestra.run.trial189.stdout:(4/10): libxslt-1.1.34-12.el9.x86_64.rpm 1.4 MB/s | 233 kB 00:00 2026-02-21T11:29:55.261 INFO:teuthology.orchestra.run.trial189.stdout:(5/10): protobuf-3.14.0-17.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2026-02-21T11:29:55.289 INFO:teuthology.orchestra.run.trial189.stdout:(6/10): socat-1.7.4.1-8.el9.x86_64.rpm 10 MB/s | 303 kB 00:00 2026-02-21T11:29:55.315 INFO:teuthology.orchestra.run.trial189.stdout:(7/10): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 64 kB 00:00 2026-02-21T11:29:55.564 INFO:teuthology.orchestra.run.trial189.stdout:(8/10): ceph-test-20.3.0-5397.gf7273f30.el9.x86 50 MB/s | 76 MB 00:01 2026-02-21T11:29:55.564 INFO:teuthology.orchestra.run.trial189.stdout:(9/10): yaml-cpp-0.6.3-5.el9.x86_64.rpm 488 kB/s | 122 kB 00:00 2026-02-21T11:29:57.048 INFO:teuthology.orchestra.run.trial189.stdout:(10/10): hwloc-libs-2.4.1-6.el9.x86_64.rpm 717 kB/s | 2.1 MB 00:02 2026-02-21T11:29:57.051 INFO:teuthology.orchestra.run.trial189.stdout:-------------------------------------------------------------------------------- 2026-02-21T11:29:57.051 INFO:teuthology.orchestra.run.trial189.stdout:Total 23 MB/s | 80 MB 00:03 2026-02-21T11:29:57.115 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction check 2026-02-21T11:29:57.126 INFO:teuthology.orchestra.run.trial189.stdout:Transaction check succeeded. 2026-02-21T11:29:57.126 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction test 2026-02-21T11:29:57.402 INFO:teuthology.orchestra.run.trial189.stdout:Transaction test succeeded. 2026-02-21T11:29:57.403 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction 2026-02-21T11:29:57.704 INFO:teuthology.orchestra.run.trial189.stdout: Preparing : 1/1 2026-02-21T11:29:57.712 INFO:teuthology.orchestra.run.trial189.stdout: Installing : yaml-cpp-0.6.3-5.el9.x86_64 1/10 2026-02-21T11:29:57.721 INFO:teuthology.orchestra.run.trial189.stdout: Installing : socat-1.7.4.1-8.el9.x86_64 2/10 2026-02-21T11:29:57.726 INFO:teuthology.orchestra.run.trial189.stdout: Installing : protobuf-3.14.0-17.el9.x86_64 3/10 2026-02-21T11:29:57.731 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libxslt-1.1.34-12.el9.x86_64 4/10 2026-02-21T11:29:57.737 INFO:teuthology.orchestra.run.trial189.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 5/10 2026-02-21T11:29:57.739 INFO:teuthology.orchestra.run.trial189.stdout: Installing : lksctp-tools-1.0.19-2.el9.x86_64 6/10 2026-02-21T11:29:57.748 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libatomic-11.5.0-14.el9.x86_64 7/10 2026-02-21T11:29:57.757 INFO:teuthology.orchestra.run.trial189.stdout: Installing : hwloc-libs-2.4.1-6.el9.x86_64 8/10 2026-02-21T11:29:57.757 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libcephsqlite-2:20.3.0-5397.gf7273f30.el9.x86_64 9/10 2026-02-21T11:30:01.112 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: libcephsqlite-2:20.3.0-5397.gf7273f30.el9.x86_64 9/10 2026-02-21T11:30:01.131 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-test-2:20.3.0-5397.gf7273f30.el9.x86_64 10/10 2026-02-21T11:30:01.478 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-test-2:20.3.0-5397.gf7273f30.el9.x86_64 10/10 2026-02-21T11:30:01.478 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-test-2:20.3.0-5397.gf7273f30.el9.x86_64 1/10 2026-02-21T11:30:01.479 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libcephsqlite-2:20.3.0-5397.gf7273f30.el9.x86_64 2/10 2026-02-21T11:30:01.479 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : hwloc-libs-2.4.1-6.el9.x86_64 3/10 2026-02-21T11:30:01.479 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libatomic-11.5.0-14.el9.x86_64 4/10 2026-02-21T11:30:01.479 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : lksctp-tools-1.0.19-2.el9.x86_64 5/10 2026-02-21T11:30:01.479 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libxslt-1.1.34-12.el9.x86_64 6/10 2026-02-21T11:30:01.479 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : protobuf-3.14.0-17.el9.x86_64 7/10 2026-02-21T11:30:01.479 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : socat-1.7.4.1-8.el9.x86_64 8/10 2026-02-21T11:30:01.479 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 9/10 2026-02-21T11:30:01.532 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : yaml-cpp-0.6.3-5.el9.x86_64 10/10 2026-02-21T11:30:01.532 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:01.532 INFO:teuthology.orchestra.run.trial189.stdout:Installed: 2026-02-21T11:30:01.533 INFO:teuthology.orchestra.run.trial189.stdout: ceph-test-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:01.533 INFO:teuthology.orchestra.run.trial189.stdout: hwloc-libs-2.4.1-6.el9.x86_64 2026-02-21T11:30:01.533 INFO:teuthology.orchestra.run.trial189.stdout: libatomic-11.5.0-14.el9.x86_64 2026-02-21T11:30:01.533 INFO:teuthology.orchestra.run.trial189.stdout: libcephsqlite-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:01.533 INFO:teuthology.orchestra.run.trial189.stdout: libxslt-1.1.34-12.el9.x86_64 2026-02-21T11:30:01.533 INFO:teuthology.orchestra.run.trial189.stdout: lksctp-tools-1.0.19-2.el9.x86_64 2026-02-21T11:30:01.533 INFO:teuthology.orchestra.run.trial189.stdout: protobuf-3.14.0-17.el9.x86_64 2026-02-21T11:30:01.533 INFO:teuthology.orchestra.run.trial189.stdout: socat-1.7.4.1-8.el9.x86_64 2026-02-21T11:30:01.533 INFO:teuthology.orchestra.run.trial189.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2026-02-21T11:30:01.533 INFO:teuthology.orchestra.run.trial189.stdout: yaml-cpp-0.6.3-5.el9.x86_64 2026-02-21T11:30:01.533 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:01.533 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:01.570 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install ceph 2026-02-21T11:30:01.871 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:22 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:01.958 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:01.960 INFO:teuthology.orchestra.run.trial189.stdout:======================================================================================= 2026-02-21T11:30:01.960 INFO:teuthology.orchestra.run.trial189.stdout: Package Arch Version Repository Size 2026-02-21T11:30:01.961 INFO:teuthology.orchestra.run.trial189.stdout:======================================================================================= 2026-02-21T11:30:01.961 INFO:teuthology.orchestra.run.trial189.stdout:Installing: 2026-02-21T11:30:01.961 INFO:teuthology.orchestra.run.trial189.stdout: ceph x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 6.5 k 2026-02-21T11:30:01.961 INFO:teuthology.orchestra.run.trial189.stdout:Installing dependencies: 2026-02-21T11:30:01.961 INFO:teuthology.orchestra.run.trial189.stdout: abseil-cpp x86_64 20211102.0-4.el9 epel 551 k 2026-02-21T11:30:01.961 INFO:teuthology.orchestra.run.trial189.stdout: ceph-grafana-dashboards noarch 2:20.3.0-5397.gf7273f30.el9 ceph-noarch 57 k 2026-02-21T11:30:01.961 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mds x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 2.4 M 2026-02-21T11:30:01.961 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mgr x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 950 k 2026-02-21T11:30:01.961 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mgr-modules-core noarch 2:20.3.0-5397.gf7273f30.el9 ceph-noarch 297 k 2026-02-21T11:30:01.961 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mon x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 5.0 M 2026-02-21T11:30:01.962 INFO:teuthology.orchestra.run.trial189.stdout: ceph-osd x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 166 k 2026-02-21T11:30:01.962 INFO:teuthology.orchestra.run.trial189.stdout: ceph-osd-classic x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 16 M 2026-02-21T11:30:01.962 INFO:teuthology.orchestra.run.trial189.stdout: ceph-osd-crimson x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 19 M 2026-02-21T11:30:01.962 INFO:teuthology.orchestra.run.trial189.stdout: ceph-prometheus-alerts noarch 2:20.3.0-5397.gf7273f30.el9 ceph-noarch 17 k 2026-02-21T11:30:01.962 INFO:teuthology.orchestra.run.trial189.stdout: cephadm noarch 2:20.3.0-5397.gf7273f30.el9 ceph-noarch 1.0 M 2026-02-21T11:30:01.962 INFO:teuthology.orchestra.run.trial189.stdout: cryptsetup x86_64 2.8.1-3.el9 baseos 351 k 2026-02-21T11:30:01.962 INFO:teuthology.orchestra.run.trial189.stdout: flexiblas x86_64 3.0.4-9.el9 appstream 30 k 2026-02-21T11:30:01.962 INFO:teuthology.orchestra.run.trial189.stdout: flexiblas-netlib x86_64 3.0.4-9.el9 appstream 3.0 M 2026-02-21T11:30:01.962 INFO:teuthology.orchestra.run.trial189.stdout: flexiblas-openblas-openmp x86_64 3.0.4-9.el9 appstream 15 k 2026-02-21T11:30:01.962 INFO:teuthology.orchestra.run.trial189.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2026-02-21T11:30:01.962 INFO:teuthology.orchestra.run.trial189.stdout: ledmon-libs x86_64 1.1.0-3.el9 baseos 40 k 2026-02-21T11:30:01.962 INFO:teuthology.orchestra.run.trial189.stdout: libconfig x86_64 1.7.2-9.el9 baseos 72 k 2026-02-21T11:30:01.963 INFO:teuthology.orchestra.run.trial189.stdout: libgfortran x86_64 11.5.0-14.el9 baseos 794 k 2026-02-21T11:30:01.963 INFO:teuthology.orchestra.run.trial189.stdout: libquadmath x86_64 11.5.0-14.el9 baseos 184 k 2026-02-21T11:30:01.963 INFO:teuthology.orchestra.run.trial189.stdout: libstoragemgmt x86_64 1.10.1-1.el9 appstream 246 k 2026-02-21T11:30:01.963 INFO:teuthology.orchestra.run.trial189.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2026-02-21T11:30:01.963 INFO:teuthology.orchestra.run.trial189.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2026-02-21T11:30:01.963 INFO:teuthology.orchestra.run.trial189.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2026-02-21T11:30:01.963 INFO:teuthology.orchestra.run.trial189.stdout: openblas x86_64 0.3.29-1.el9 appstream 42 k 2026-02-21T11:30:01.963 INFO:teuthology.orchestra.run.trial189.stdout: openblas-openmp x86_64 0.3.29-1.el9 appstream 5.3 M 2026-02-21T11:30:01.963 INFO:teuthology.orchestra.run.trial189.stdout: protobuf-compiler x86_64 3.14.0-17.el9 crb 862 k 2026-02-21T11:30:01.963 INFO:teuthology.orchestra.run.trial189.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2026-02-21T11:30:01.963 INFO:teuthology.orchestra.run.trial189.stdout: python3-autocommand noarch 2.2.2-8.el9 epel 29 k 2026-02-21T11:30:01.964 INFO:teuthology.orchestra.run.trial189.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2026-02-21T11:30:01.964 INFO:teuthology.orchestra.run.trial189.stdout: python3-backports-tarfile noarch 1.2.0-1.el9 epel 60 k 2026-02-21T11:30:01.964 INFO:teuthology.orchestra.run.trial189.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2026-02-21T11:30:01.964 INFO:teuthology.orchestra.run.trial189.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2026-02-21T11:30:01.964 INFO:teuthology.orchestra.run.trial189.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2026-02-21T11:30:01.964 INFO:teuthology.orchestra.run.trial189.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2026-02-21T11:30:01.964 INFO:teuthology.orchestra.run.trial189.stdout: python3-cheroot noarch 10.0.1-4.el9 epel 173 k 2026-02-21T11:30:01.964 INFO:teuthology.orchestra.run.trial189.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2026-02-21T11:30:01.964 INFO:teuthology.orchestra.run.trial189.stdout: python3-cryptography x86_64 36.0.1-5.el9 baseos 1.2 M 2026-02-21T11:30:01.964 INFO:teuthology.orchestra.run.trial189.stdout: python3-devel x86_64 3.9.25-3.el9 appstream 244 k 2026-02-21T11:30:01.964 INFO:teuthology.orchestra.run.trial189.stdout: python3-google-auth noarch 1:2.45.0-1.el9 epel 254 k 2026-02-21T11:30:01.964 INFO:teuthology.orchestra.run.trial189.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2026-02-21T11:30:01.964 INFO:teuthology.orchestra.run.trial189.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2026-02-21T11:30:01.965 INFO:teuthology.orchestra.run.trial189.stdout: python3-isodate noarch 0.6.1-3.el9 epel 56 k 2026-02-21T11:30:01.965 INFO:teuthology.orchestra.run.trial189.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2026-02-21T11:30:01.965 INFO:teuthology.orchestra.run.trial189.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2026-02-21T11:30:01.965 INFO:teuthology.orchestra.run.trial189.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2026-02-21T11:30:01.965 INFO:teuthology.orchestra.run.trial189.stdout: python3-jaraco-context noarch 6.0.1-3.el9 epel 20 k 2026-02-21T11:30:01.965 INFO:teuthology.orchestra.run.trial189.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2026-02-21T11:30:01.965 INFO:teuthology.orchestra.run.trial189.stdout: python3-jaraco-text noarch 4.0.0-2.el9 epel 26 k 2026-02-21T11:30:01.965 INFO:teuthology.orchestra.run.trial189.stdout: python3-jinja2 noarch 2.11.3-8.el9 appstream 249 k 2026-02-21T11:30:01.965 INFO:teuthology.orchestra.run.trial189.stdout: python3-jmespath noarch 1.0.1-1.el9 appstream 48 k 2026-02-21T11:30:01.965 INFO:teuthology.orchestra.run.trial189.stdout: python3-kubernetes noarch 1:26.1.0-3.el9 epel 1.0 M 2026-02-21T11:30:01.965 INFO:teuthology.orchestra.run.trial189.stdout: python3-libstoragemgmt x86_64 1.10.1-1.el9 appstream 177 k 2026-02-21T11:30:01.965 INFO:teuthology.orchestra.run.trial189.stdout: python3-lxml x86_64 4.6.5-3.el9 appstream 1.2 M 2026-02-21T11:30:01.965 INFO:teuthology.orchestra.run.trial189.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2026-02-21T11:30:01.966 INFO:teuthology.orchestra.run.trial189.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2026-02-21T11:30:01.966 INFO:teuthology.orchestra.run.trial189.stdout: python3-msgpack x86_64 1.0.3-2.el9 epel 86 k 2026-02-21T11:30:01.966 INFO:teuthology.orchestra.run.trial189.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2026-02-21T11:30:01.966 INFO:teuthology.orchestra.run.trial189.stdout: python3-numpy x86_64 1:1.23.5-2.el9 appstream 6.1 M 2026-02-21T11:30:01.966 INFO:teuthology.orchestra.run.trial189.stdout: python3-numpy-f2py x86_64 1:1.23.5-2.el9 appstream 442 k 2026-02-21T11:30:01.966 INFO:teuthology.orchestra.run.trial189.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2026-02-21T11:30:01.966 INFO:teuthology.orchestra.run.trial189.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2026-02-21T11:30:01.966 INFO:teuthology.orchestra.run.trial189.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2026-02-21T11:30:01.966 INFO:teuthology.orchestra.run.trial189.stdout: python3-protobuf noarch 3.14.0-17.el9 appstream 267 k 2026-02-21T11:30:01.966 INFO:teuthology.orchestra.run.trial189.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2026-02-21T11:30:01.966 INFO:teuthology.orchestra.run.trial189.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2026-02-21T11:30:01.966 INFO:teuthology.orchestra.run.trial189.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2026-02-21T11:30:01.967 INFO:teuthology.orchestra.run.trial189.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2026-02-21T11:30:01.967 INFO:teuthology.orchestra.run.trial189.stdout: python3-pyparsing noarch 2.4.7-9.el9 baseos 150 k 2026-02-21T11:30:01.967 INFO:teuthology.orchestra.run.trial189.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2026-02-21T11:30:01.967 INFO:teuthology.orchestra.run.trial189.stdout: python3-requests noarch 2.25.1-10.el9 baseos 126 k 2026-02-21T11:30:01.967 INFO:teuthology.orchestra.run.trial189.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2026-02-21T11:30:01.967 INFO:teuthology.orchestra.run.trial189.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2026-02-21T11:30:01.967 INFO:teuthology.orchestra.run.trial189.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2026-02-21T11:30:01.967 INFO:teuthology.orchestra.run.trial189.stdout: python3-scipy x86_64 1.9.3-2.el9 appstream 19 M 2026-02-21T11:30:01.967 INFO:teuthology.orchestra.run.trial189.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2026-02-21T11:30:01.967 INFO:teuthology.orchestra.run.trial189.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2026-02-21T11:30:01.967 INFO:teuthology.orchestra.run.trial189.stdout: python3-typing-extensions noarch 4.15.0-1.el9 epel 86 k 2026-02-21T11:30:01.967 INFO:teuthology.orchestra.run.trial189.stdout: python3-urllib3 noarch 1.26.5-7.el9 baseos 218 k 2026-02-21T11:30:01.968 INFO:teuthology.orchestra.run.trial189.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2026-02-21T11:30:01.968 INFO:teuthology.orchestra.run.trial189.stdout: python3-xmlsec x86_64 1.3.13-1.el9 epel 48 k 2026-02-21T11:30:01.968 INFO:teuthology.orchestra.run.trial189.stdout: python3-xmltodict noarch 0.12.0-15.el9 epel 22 k 2026-02-21T11:30:01.968 INFO:teuthology.orchestra.run.trial189.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2026-02-21T11:30:01.968 INFO:teuthology.orchestra.run.trial189.stdout: unzip x86_64 6.0-59.el9 baseos 182 k 2026-02-21T11:30:01.968 INFO:teuthology.orchestra.run.trial189.stdout: xmlsec1 x86_64 1.2.29-13.el9 appstream 189 k 2026-02-21T11:30:01.968 INFO:teuthology.orchestra.run.trial189.stdout: xmlsec1-openssl x86_64 1.2.29-13.el9 appstream 90 k 2026-02-21T11:30:01.968 INFO:teuthology.orchestra.run.trial189.stdout: zip x86_64 3.0-35.el9 baseos 266 k 2026-02-21T11:30:01.968 INFO:teuthology.orchestra.run.trial189.stdout:Installing weak dependencies: 2026-02-21T11:30:01.968 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mgr-cephadm noarch 2:20.3.0-5397.gf7273f30.el9 ceph-noarch 184 k 2026-02-21T11:30:01.968 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mgr-dashboard noarch 2:20.3.0-5397.gf7273f30.el9 ceph-noarch 51 M 2026-02-21T11:30:01.968 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mgr-diskprediction-local noarch 2:20.3.0-5397.gf7273f30.el9 ceph-noarch 7.4 M 2026-02-21T11:30:01.969 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mgr-k8sevents noarch 2:20.3.0-5397.gf7273f30.el9 ceph-noarch 22 k 2026-02-21T11:30:01.969 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mgr-rook noarch 2:20.3.0-5397.gf7273f30.el9 ceph-noarch 50 k 2026-02-21T11:30:01.969 INFO:teuthology.orchestra.run.trial189.stdout: ceph-volume noarch 2:20.3.0-5397.gf7273f30.el9 ceph-noarch 301 k 2026-02-21T11:30:01.969 INFO:teuthology.orchestra.run.trial189.stdout: python3-influxdb noarch 5.3.1-1.el9 epel 139 k 2026-02-21T11:30:01.969 INFO:teuthology.orchestra.run.trial189.stdout: python3-saml noarch 1.16.0-1.el9 epel 125 k 2026-02-21T11:30:01.969 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:01.969 INFO:teuthology.orchestra.run.trial189.stdout:Transaction Summary 2026-02-21T11:30:01.969 INFO:teuthology.orchestra.run.trial189.stdout:======================================================================================= 2026-02-21T11:30:01.969 INFO:teuthology.orchestra.run.trial189.stdout:Install 96 Packages 2026-02-21T11:30:01.969 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:01.970 INFO:teuthology.orchestra.run.trial189.stdout:Total download size: 158 M 2026-02-21T11:30:01.970 INFO:teuthology.orchestra.run.trial189.stdout:Installed size: 732 M 2026-02-21T11:30:01.970 INFO:teuthology.orchestra.run.trial189.stdout:Downloading Packages: 2026-02-21T11:30:02.284 INFO:teuthology.orchestra.run.trial189.stdout:(1/96): ceph-20.3.0-5397.gf7273f30.el9.x86_64.r 124 kB/s | 6.5 kB 00:00 2026-02-21T11:30:02.373 INFO:teuthology.orchestra.run.trial189.stdout:(2/96): ceph-mgr-20.3.0-5397.gf7273f30.el9.x86_ 6.6 MB/s | 950 kB 00:00 2026-02-21T11:30:02.400 INFO:teuthology.orchestra.run.trial189.stdout:(3/96): ceph-mds-20.3.0-5397.gf7273f30.el9.x86_ 14 MB/s | 2.4 MB 00:00 2026-02-21T11:30:02.403 INFO:teuthology.orchestra.run.trial189.stdout:(4/96): ceph-osd-20.3.0-5397.gf7273f30.el9.x86_ 5.6 MB/s | 166 kB 00:00 2026-02-21T11:30:02.583 INFO:teuthology.orchestra.run.trial189.stdout:(5/96): ceph-mon-20.3.0-5397.gf7273f30.el9.x86_ 17 MB/s | 5.0 MB 00:00 2026-02-21T11:30:02.630 INFO:teuthology.orchestra.run.trial189.stdout:(6/96): ceph-grafana-dashboards-20.3.0-5397.gf7 1.2 MB/s | 57 kB 00:00 2026-02-21T11:30:02.682 INFO:teuthology.orchestra.run.trial189.stdout:(7/96): ceph-mgr-cephadm-20.3.0-5397.gf7273f30. 3.5 MB/s | 184 kB 00:00 2026-02-21T11:30:03.025 INFO:teuthology.orchestra.run.trial189.stdout:(8/96): ceph-osd-classic-20.3.0-5397.gf7273f30. 25 MB/s | 16 MB 00:00 2026-02-21T11:30:03.355 INFO:teuthology.orchestra.run.trial189.stdout:(9/96): ceph-mgr-diskprediction-local-20.3.0-53 22 MB/s | 7.4 MB 00:00 2026-02-21T11:30:03.393 INFO:teuthology.orchestra.run.trial189.stdout:(10/96): ceph-osd-crimson-20.3.0-5397.gf7273f30 19 MB/s | 19 MB 00:00 2026-02-21T11:30:03.393 INFO:teuthology.orchestra.run.trial189.stdout:(11/96): ceph-mgr-k8sevents-20.3.0-5397.gf7273f 586 kB/s | 22 kB 00:00 2026-02-21T11:30:03.408 INFO:teuthology.orchestra.run.trial189.stdout:(12/96): ceph-mgr-modules-core-20.3.0-5397.gf72 19 MB/s | 297 kB 00:00 2026-02-21T11:30:03.413 INFO:teuthology.orchestra.run.trial189.stdout:(13/96): ceph-mgr-rook-20.3.0-5397.gf7273f30.el 2.5 MB/s | 50 kB 00:00 2026-02-21T11:30:03.425 INFO:teuthology.orchestra.run.trial189.stdout:(14/96): ceph-prometheus-alerts-20.3.0-5397.gf7 1.0 MB/s | 17 kB 00:00 2026-02-21T11:30:03.434 INFO:teuthology.orchestra.run.trial189.stdout:(15/96): ceph-volume-20.3.0-5397.gf7273f30.el9. 14 MB/s | 301 kB 00:00 2026-02-21T11:30:03.478 INFO:teuthology.orchestra.run.trial189.stdout:(16/96): cephadm-20.3.0-5397.gf7273f30.el9.noar 19 MB/s | 1.0 MB 00:00 2026-02-21T11:30:04.054 INFO:teuthology.orchestra.run.trial189.stdout:(17/96): ledmon-libs-1.1.0-3.el9.x86_64.rpm 70 kB/s | 40 kB 00:00 2026-02-21T11:30:04.160 INFO:teuthology.orchestra.run.trial189.stdout:(18/96): ceph-mgr-dashboard-20.3.0-5397.gf7273f 34 MB/s | 51 MB 00:01 2026-02-21T11:30:04.262 INFO:teuthology.orchestra.run.trial189.stdout:(19/96): cryptsetup-2.8.1-3.el9.x86_64.rpm 424 kB/s | 351 kB 00:00 2026-02-21T11:30:04.717 INFO:teuthology.orchestra.run.trial189.stdout:(20/96): libconfig-1.7.2-9.el9.x86_64.rpm 109 kB/s | 72 kB 00:00 2026-02-21T11:30:04.997 INFO:teuthology.orchestra.run.trial189.stdout:(21/96): libquadmath-11.5.0-14.el9.x86_64.rpm 251 kB/s | 184 kB 00:00 2026-02-21T11:30:05.054 INFO:teuthology.orchestra.run.trial189.stdout:(22/96): libgfortran-11.5.0-14.el9.x86_64.rpm 889 kB/s | 794 kB 00:00 2026-02-21T11:30:05.483 INFO:teuthology.orchestra.run.trial189.stdout:(23/96): python3-cffi-1.14.5-5.el9.x86_64.rpm 331 kB/s | 253 kB 00:00 2026-02-21T11:30:05.722 INFO:teuthology.orchestra.run.trial189.stdout:(24/96): python3-ply-3.11-14.el9.noarch.rpm 159 kB/s | 106 kB 00:00 2026-02-21T11:30:05.940 INFO:teuthology.orchestra.run.trial189.stdout:(25/96): python3-cryptography-36.0.1-5.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2026-02-21T11:30:06.183 INFO:teuthology.orchestra.run.trial189.stdout:(26/96): python3-pycparser-2.20-6.el9.noarch.rp 193 kB/s | 135 kB 00:00 2026-02-21T11:30:06.433 INFO:teuthology.orchestra.run.trial189.stdout:(27/96): python3-pyparsing-2.4.7-9.el9.noarch.r 212 kB/s | 150 kB 00:00 2026-02-21T11:30:06.634 INFO:teuthology.orchestra.run.trial189.stdout:(28/96): python3-requests-2.25.1-10.el9.noarch. 182 kB/s | 126 kB 00:00 2026-02-21T11:30:06.930 INFO:teuthology.orchestra.run.trial189.stdout:(29/96): python3-urllib3-1.26.5-7.el9.noarch.rp 291 kB/s | 218 kB 00:00 2026-02-21T11:30:07.168 INFO:teuthology.orchestra.run.trial189.stdout:(30/96): unzip-6.0-59.el9.x86_64.rpm 247 kB/s | 182 kB 00:00 2026-02-21T11:30:07.409 INFO:teuthology.orchestra.run.trial189.stdout:(31/96): zip-3.0-35.el9.x86_64.rpm 343 kB/s | 266 kB 00:00 2026-02-21T11:30:07.487 INFO:teuthology.orchestra.run.trial189.stdout:(32/96): flexiblas-3.0.4-9.el9.x86_64.rpm 53 kB/s | 30 kB 00:00 2026-02-21T11:30:07.890 INFO:teuthology.orchestra.run.trial189.stdout:(33/96): flexiblas-openblas-openmp-3.0.4-9.el9. 31 kB/s | 15 kB 00:00 2026-02-21T11:30:08.232 INFO:teuthology.orchestra.run.trial189.stdout:(34/96): flexiblas-netlib-3.0.4-9.el9.x86_64.rp 2.8 MB/s | 3.0 MB 00:01 2026-02-21T11:30:08.251 INFO:teuthology.orchestra.run.trial189.stdout:(35/96): libstoragemgmt-1.10.1-1.el9.x86_64.rpm 323 kB/s | 246 kB 00:00 2026-02-21T11:30:08.625 INFO:teuthology.orchestra.run.trial189.stdout:(36/96): lua-5.4.4-4.el9.x86_64.rpm 256 kB/s | 188 kB 00:00 2026-02-21T11:30:08.811 INFO:teuthology.orchestra.run.trial189.stdout:(37/96): openblas-0.3.29-1.el9.x86_64.rpm 73 kB/s | 42 kB 00:00 2026-02-21T11:30:09.358 INFO:teuthology.orchestra.run.trial189.stdout:(38/96): openblas-openmp-0.3.29-1.el9.x86_64.rp 4.8 MB/s | 5.3 MB 00:01 2026-02-21T11:30:09.579 INFO:teuthology.orchestra.run.trial189.stdout:(39/96): python3-devel-3.9.25-3.el9.x86_64.rpm 318 kB/s | 244 kB 00:00 2026-02-21T11:30:09.747 INFO:teuthology.orchestra.run.trial189.stdout:(40/96): python3-babel-2.9.1-2.el9.noarch.rpm 5.3 MB/s | 6.0 MB 00:01 2026-02-21T11:30:10.126 INFO:teuthology.orchestra.run.trial189.stdout:(41/96): python3-jinja2-2.11.3-8.el9.noarch.rpm 324 kB/s | 249 kB 00:00 2026-02-21T11:30:10.170 INFO:teuthology.orchestra.run.trial189.stdout:(42/96): python3-jmespath-1.0.1-1.el9.noarch.rp 81 kB/s | 48 kB 00:00 2026-02-21T11:30:10.478 INFO:teuthology.orchestra.run.trial189.stdout:(43/96): python3-libstoragemgmt-1.10.1-1.el9.x8 242 kB/s | 177 kB 00:00 2026-02-21T11:30:10.737 INFO:teuthology.orchestra.run.trial189.stdout:(44/96): python3-markupsafe-1.1.1-12.el9.x86_64 61 kB/s | 35 kB 00:00 2026-02-21T11:30:11.076 INFO:teuthology.orchestra.run.trial189.stdout:(45/96): python3-lxml-4.6.5-3.el9.x86_64.rpm 1.3 MB/s | 1.2 MB 00:00 2026-02-21T11:30:11.564 INFO:teuthology.orchestra.run.trial189.stdout:(46/96): python3-numpy-f2py-1.23.5-2.el9.x86_64 534 kB/s | 442 kB 00:00 2026-02-21T11:30:11.606 INFO:teuthology.orchestra.run.trial189.stdout:(47/96): python3-numpy-1.23.5-2.el9.x86_64.rpm 5.4 MB/s | 6.1 MB 00:01 2026-02-21T11:30:11.715 INFO:teuthology.orchestra.run.trial189.stdout:(48/96): python3-packaging-20.9-5.el9.noarch.rp 121 kB/s | 77 kB 00:00 2026-02-21T11:30:12.328 INFO:teuthology.orchestra.run.trial189.stdout:(49/96): python3-pyasn1-0.4.8-6.el9.noarch.rpm 220 kB/s | 159 kB 00:00 2026-02-21T11:30:12.339 INFO:teuthology.orchestra.run.trial189.stdout:(50/96): python3-protobuf-3.14.0-17.el9.noarch. 345 kB/s | 267 kB 00:00 2026-02-21T11:30:12.495 INFO:teuthology.orchestra.run.trial189.stdout:(51/96): python3-pyasn1-modules-0.4.8-6.el9.noa 358 kB/s | 279 kB 00:00 2026-02-21T11:30:12.927 INFO:teuthology.orchestra.run.trial189.stdout:(52/96): python3-requests-oauthlib-1.3.0-12.el9 90 kB/s | 54 kB 00:00 2026-02-21T11:30:13.073 INFO:teuthology.orchestra.run.trial189.stdout:(53/96): python3-toml-0.10.2-6.el9.noarch.rpm 72 kB/s | 42 kB 00:00 2026-02-21T11:30:13.662 INFO:teuthology.orchestra.run.trial189.stdout:(54/96): xmlsec1-1.2.29-13.el9.x86_64.rpm 257 kB/s | 189 kB 00:00 2026-02-21T11:30:13.677 INFO:teuthology.orchestra.run.trial189.stdout:(55/96): lua-devel-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 22 kB 00:00 2026-02-21T11:30:13.696 INFO:teuthology.orchestra.run.trial189.stdout:(56/96): protobuf-compiler-3.14.0-17.el9.x86_64 44 MB/s | 862 kB 00:00 2026-02-21T11:30:13.748 INFO:teuthology.orchestra.run.trial189.stdout:(57/96): python3-scipy-1.9.3-2.el9.x86_64.rpm 14 MB/s | 19 MB 00:01 2026-02-21T11:30:13.752 INFO:teuthology.orchestra.run.trial189.stdout:(58/96): xmlsec1-openssl-1.2.29-13.el9.x86_64.r 133 kB/s | 90 kB 00:00 2026-02-21T11:30:13.832 INFO:teuthology.orchestra.run.trial189.stdout:(59/96): grpc-data-1.46.7-10.el9.noarch.rpm 239 kB/s | 19 kB 00:00 2026-02-21T11:30:13.879 INFO:teuthology.orchestra.run.trial189.stdout:(60/96): luarocks-3.9.2-5.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2026-02-21T11:30:13.899 INFO:teuthology.orchestra.run.trial189.stdout:(61/96): python3-autocommand-2.2.2-8.el9.noarch 1.4 MB/s | 29 kB 00:00 2026-02-21T11:30:13.910 INFO:teuthology.orchestra.run.trial189.stdout:(62/96): abseil-cpp-20211102.0-4.el9.x86_64.rpm 2.5 MB/s | 551 kB 00:00 2026-02-21T11:30:13.923 INFO:teuthology.orchestra.run.trial189.stdout:(63/96): python3-backports-tarfile-1.2.0-1.el9. 2.5 MB/s | 60 kB 00:00 2026-02-21T11:30:13.931 INFO:teuthology.orchestra.run.trial189.stdout:(64/96): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 2.1 MB/s | 43 kB 00:00 2026-02-21T11:30:13.932 INFO:teuthology.orchestra.run.trial189.stdout:(65/96): python3-asyncssh-2.13.2-5.el9.noarch.r 5.4 MB/s | 548 kB 00:00 2026-02-21T11:30:13.943 INFO:teuthology.orchestra.run.trial189.stdout:(66/96): python3-cachetools-4.2.4-1.el9.noarch. 1.5 MB/s | 32 kB 00:00 2026-02-21T11:30:13.951 INFO:teuthology.orchestra.run.trial189.stdout:(67/96): python3-certifi-2023.05.07-4.el9.noarc 681 kB/s | 14 kB 00:00 2026-02-21T11:30:13.954 INFO:teuthology.orchestra.run.trial189.stdout:(68/96): python3-cheroot-10.0.1-4.el9.noarch.rp 7.6 MB/s | 173 kB 00:00 2026-02-21T11:30:13.977 INFO:teuthology.orchestra.run.trial189.stdout:(69/96): python3-google-auth-2.45.0-1.el9.noarc 9.9 MB/s | 254 kB 00:00 2026-02-21T11:30:13.992 INFO:teuthology.orchestra.run.trial189.stdout:(70/96): python3-cherrypy-18.6.1-2.el9.noarch.r 7.2 MB/s | 358 kB 00:00 2026-02-21T11:30:14.002 INFO:teuthology.orchestra.run.trial189.stdout:(71/96): python3-grpcio-tools-1.46.7-10.el9.x86 5.7 MB/s | 144 kB 00:00 2026-02-21T11:30:14.016 INFO:teuthology.orchestra.run.trial189.stdout:(72/96): python3-grpcio-1.46.7-10.el9.x86_64.rp 33 MB/s | 2.0 MB 00:00 2026-02-21T11:30:14.016 INFO:teuthology.orchestra.run.trial189.stdout:(73/96): python3-influxdb-5.3.1-1.el9.noarch.rp 5.6 MB/s | 139 kB 00:00 2026-02-21T11:30:14.023 INFO:teuthology.orchestra.run.trial189.stdout:(74/96): python3-isodate-0.6.1-3.el9.noarch.rpm 2.6 MB/s | 56 kB 00:00 2026-02-21T11:30:14.035 INFO:teuthology.orchestra.run.trial189.stdout:(75/96): python3-jaraco-8.2.1-3.el9.noarch.rpm 557 kB/s | 11 kB 00:00 2026-02-21T11:30:14.035 INFO:teuthology.orchestra.run.trial189.stdout:(76/96): python3-jaraco-classes-3.2.1-5.el9.noa 921 kB/s | 18 kB 00:00 2026-02-21T11:30:14.044 INFO:teuthology.orchestra.run.trial189.stdout:(77/96): python3-jaraco-collections-3.0.0-8.el9 1.1 MB/s | 23 kB 00:00 2026-02-21T11:30:14.054 INFO:teuthology.orchestra.run.trial189.stdout:(78/96): python3-jaraco-context-6.0.1-3.el9.noa 1.0 MB/s | 20 kB 00:00 2026-02-21T11:30:14.055 INFO:teuthology.orchestra.run.trial189.stdout:(79/96): python3-jaraco-functools-3.5.0-2.el9.n 1.0 MB/s | 19 kB 00:00 2026-02-21T11:30:14.066 INFO:teuthology.orchestra.run.trial189.stdout:(80/96): python3-jaraco-text-4.0.0-2.el9.noarch 1.2 MB/s | 26 kB 00:00 2026-02-21T11:30:14.076 INFO:teuthology.orchestra.run.trial189.stdout:(81/96): python3-more-itertools-8.12.0-2.el9.no 3.6 MB/s | 79 kB 00:00 2026-02-21T11:30:14.080 INFO:teuthology.orchestra.run.trial189.stdout:(82/96): python3-kubernetes-26.1.0-3.el9.noarch 40 MB/s | 1.0 MB 00:00 2026-02-21T11:30:14.088 INFO:teuthology.orchestra.run.trial189.stdout:(83/96): python3-msgpack-1.0.3-2.el9.x86_64.rpm 3.7 MB/s | 86 kB 00:00 2026-02-21T11:30:14.096 INFO:teuthology.orchestra.run.trial189.stdout:(84/96): python3-natsort-7.1.1-5.el9.noarch.rpm 2.8 MB/s | 58 kB 00:00 2026-02-21T11:30:14.099 INFO:teuthology.orchestra.run.trial189.stdout:(85/96): python3-portend-3.1.0-2.el9.noarch.rpm 866 kB/s | 16 kB 00:00 2026-02-21T11:30:14.113 INFO:teuthology.orchestra.run.trial189.stdout:(86/96): python3-pyOpenSSL-21.0.0-1.el9.noarch. 3.7 MB/s | 90 kB 00:00 2026-02-21T11:30:14.115 INFO:teuthology.orchestra.run.trial189.stdout:(87/96): python3-repoze-lru-0.7-16.el9.noarch.r 1.6 MB/s | 31 kB 00:00 2026-02-21T11:30:14.119 INFO:teuthology.orchestra.run.trial189.stdout:(88/96): python3-routes-2.5.1-5.el9.noarch.rpm 9.3 MB/s | 188 kB 00:00 2026-02-21T11:30:14.135 INFO:teuthology.orchestra.run.trial189.stdout:(89/96): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2026-02-21T11:30:14.137 INFO:teuthology.orchestra.run.trial189.stdout:(90/96): python3-saml-1.16.0-1.el9.noarch.rpm 5.6 MB/s | 125 kB 00:00 2026-02-21T11:30:14.138 INFO:teuthology.orchestra.run.trial189.stdout:(91/96): python3-tempora-5.0.0-2.el9.noarch.rpm 1.8 MB/s | 36 kB 00:00 2026-02-21T11:30:14.157 INFO:teuthology.orchestra.run.trial189.stdout:(92/96): python3-xmlsec-1.3.13-1.el9.x86_64.rpm 2.4 MB/s | 48 kB 00:00 2026-02-21T11:30:14.158 INFO:teuthology.orchestra.run.trial189.stdout:(93/96): python3-websocket-client-1.2.3-2.el9.n 4.2 MB/s | 90 kB 00:00 2026-02-21T11:30:14.159 INFO:teuthology.orchestra.run.trial189.stdout:(94/96): python3-typing-extensions-4.15.0-1.el9 3.6 MB/s | 86 kB 00:00 2026-02-21T11:30:14.177 INFO:teuthology.orchestra.run.trial189.stdout:(95/96): python3-xmltodict-0.12.0-15.el9.noarch 1.1 MB/s | 22 kB 00:00 2026-02-21T11:30:14.178 INFO:teuthology.orchestra.run.trial189.stdout:(96/96): python3-zc-lockfile-2.0-10.el9.noarch. 982 kB/s | 20 kB 00:00 2026-02-21T11:30:14.183 INFO:teuthology.orchestra.run.trial189.stdout:-------------------------------------------------------------------------------- 2026-02-21T11:30:14.183 INFO:teuthology.orchestra.run.trial189.stdout:Total 13 MB/s | 158 MB 00:12 2026-02-21T11:30:14.603 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction check 2026-02-21T11:30:14.641 INFO:teuthology.orchestra.run.trial189.stdout:Transaction check succeeded. 2026-02-21T11:30:14.641 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction test 2026-02-21T11:30:15.246 INFO:teuthology.orchestra.run.trial189.stdout:Transaction test succeeded. 2026-02-21T11:30:15.246 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction 2026-02-21T11:30:15.887 INFO:teuthology.orchestra.run.trial189.stdout: Preparing : 1/1 2026-02-21T11:30:15.890 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/96 2026-02-21T11:30:15.894 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/96 2026-02-21T11:30:15.915 INFO:teuthology.orchestra.run.trial189.stdout: Installing : xmlsec1-1.2.29-13.el9.x86_64 3/96 2026-02-21T11:30:15.949 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 4/96 2026-02-21T11:30:15.956 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-lxml-4.6.5-3.el9.x86_64 5/96 2026-02-21T11:30:15.958 INFO:teuthology.orchestra.run.trial189.stdout: Installing : lua-5.4.4-4.el9.x86_64 6/96 2026-02-21T11:30:15.963 INFO:teuthology.orchestra.run.trial189.stdout: Installing : flexiblas-3.0.4-9.el9.x86_64 7/96 2026-02-21T11:30:15.986 INFO:teuthology.orchestra.run.trial189.stdout: Installing : unzip-6.0-59.el9.x86_64 8/96 2026-02-21T11:30:15.999 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-urllib3-1.26.5-7.el9.noarch 9/96 2026-02-21T11:30:16.002 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-requests-2.25.1-10.el9.noarch 10/96 2026-02-21T11:30:16.008 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libquadmath-11.5.0-14.el9.x86_64 11/96 2026-02-21T11:30:16.010 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libgfortran-11.5.0-14.el9.x86_64 12/96 2026-02-21T11:30:16.021 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ledmon-libs-1.1.0-3.el9.x86_64 13/96 2026-02-21T11:30:16.028 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 14/96 2026-02-21T11:30:16.052 INFO:teuthology.orchestra.run.trial189.stdout: Installing : zip-3.0-35.el9.x86_64 15/96 2026-02-21T11:30:16.055 INFO:teuthology.orchestra.run.trial189.stdout: Installing : luarocks-3.9.2-5.el9.noarch 16/96 2026-02-21T11:30:16.106 INFO:teuthology.orchestra.run.trial189.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 17/96 2026-02-21T11:30:16.118 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 18/96 2026-02-21T11:30:16.121 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-rsa-4.9-2.el9.noarch 19/96 2026-02-21T11:30:16.125 INFO:teuthology.orchestra.run.trial189.stdout: Installing : xmlsec1-openssl-1.2.29-13.el9.x86_64 20/96 2026-02-21T11:30:16.130 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-xmlsec-1.3.13-1.el9.x86_64 21/96 2026-02-21T11:30:16.135 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 22/96 2026-02-21T11:30:16.138 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 23/96 2026-02-21T11:30:16.153 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-xmltodict-0.12.0-15.el9.noarch 24/96 2026-02-21T11:30:16.157 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 25/96 2026-02-21T11:30:16.162 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-typing-extensions-4.15.0-1.el9.noarch 26/96 2026-02-21T11:30:16.174 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 27/96 2026-02-21T11:30:16.183 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 28/96 2026-02-21T11:30:16.189 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 29/96 2026-02-21T11:30:16.208 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-msgpack-1.0.3-2.el9.x86_64 30/96 2026-02-21T11:30:16.219 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-influxdb-5.3.1-1.el9.noarch 31/96 2026-02-21T11:30:16.235 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-isodate-0.6.1-3.el9.noarch 32/96 2026-02-21T11:30:16.240 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-saml-1.16.0-1.el9.noarch 33/96 2026-02-21T11:30:16.247 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 34/96 2026-02-21T11:30:16.287 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 35/96 2026-02-21T11:30:16.613 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-google-auth-1:2.45.0-1.el9.noarch 36/96 2026-02-21T11:30:16.623 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-kubernetes-1:26.1.0-3.el9.noarch 37/96 2026-02-21T11:30:16.627 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-backports-tarfile-1.2.0-1.el9.noarch 38/96 2026-02-21T11:30:16.633 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-jaraco-context-6.0.1-3.el9.noarch 39/96 2026-02-21T11:30:16.635 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-autocommand-2.2.2-8.el9.noarch 40/96 2026-02-21T11:30:16.662 INFO:teuthology.orchestra.run.trial189.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 41/96 2026-02-21T11:30:16.707 INFO:teuthology.orchestra.run.trial189.stdout: Installing : abseil-cpp-20211102.0-4.el9.x86_64 42/96 2026-02-21T11:30:16.717 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 43/96 2026-02-21T11:30:16.722 INFO:teuthology.orchestra.run.trial189.stdout: Installing : protobuf-compiler-3.14.0-17.el9.x86_64 44/96 2026-02-21T11:30:16.726 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 45/96 2026-02-21T11:30:16.731 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 46/96 2026-02-21T11:30:16.735 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-jaraco-text-4.0.0-2.el9.noarch 47/96 2026-02-21T11:30:16.742 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 48/96 2026-02-21T11:30:16.745 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 49/96 2026-02-21T11:30:16.773 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 50/96 2026-02-21T11:30:16.783 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-protobuf-3.14.0-17.el9.noarch 51/96 2026-02-21T11:30:16.789 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 52/96 2026-02-21T11:30:16.796 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 53/96 2026-02-21T11:30:16.830 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-jmespath-1.0.1-1.el9.noarch 54/96 2026-02-21T11:30:17.054 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-devel-3.9.25-3.el9.x86_64 55/96 2026-02-21T11:30:17.078 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 56/96 2026-02-21T11:30:17.082 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-jinja2-2.11.3-8.el9.noarch 57/96 2026-02-21T11:30:17.141 INFO:teuthology.orchestra.run.trial189.stdout: Installing : openblas-0.3.29-1.el9.x86_64 58/96 2026-02-21T11:30:17.143 INFO:teuthology.orchestra.run.trial189.stdout: Installing : openblas-openmp-0.3.29-1.el9.x86_64 59/96 2026-02-21T11:30:17.164 INFO:teuthology.orchestra.run.trial189.stdout: Installing : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 60/96 2026-02-21T11:30:17.490 INFO:teuthology.orchestra.run.trial189.stdout: Installing : flexiblas-netlib-3.0.4-9.el9.x86_64 61/96 2026-02-21T11:30:17.563 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-numpy-1:1.23.5-2.el9.x86_64 62/96 2026-02-21T11:30:18.236 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 63/96 2026-02-21T11:30:18.253 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-scipy-1.9.3-2.el9.x86_64 64/96 2026-02-21T11:30:18.262 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-pyparsing-2.4.7-9.el9.noarch 65/96 2026-02-21T11:30:18.269 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-packaging-20.9-5.el9.noarch 66/96 2026-02-21T11:30:18.282 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-ply-3.11-14.el9.noarch 67/96 2026-02-21T11:30:18.298 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 68/96 2026-02-21T11:30:18.374 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 69/96 2026-02-21T11:30:18.383 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-cryptography-36.0.1-5.el9.x86_64 70/96 2026-02-21T11:30:18.406 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 71/96 2026-02-21T11:30:18.436 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-cheroot-10.0.1-4.el9.noarch 72/96 2026-02-21T11:30:18.487 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 73/96 2026-02-21T11:30:18.493 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 74/96 2026-02-21T11:30:18.497 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 75/96 2026-02-21T11:30:18.498 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libconfig-1.7.2-9.el9.x86_64 76/96 2026-02-21T11:30:18.515 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-21T11:30:18.516 INFO:teuthology.orchestra.run.trial189.stdout:Creating group 'libstoragemgmt' with GID 991. 2026-02-21T11:30:18.516 INFO:teuthology.orchestra.run.trial189.stdout:Creating user 'libstoragemgmt' (daemon account for libstoragemgmt) with UID 991 and GID 991. 2026-02-21T11:30:18.516 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:18.523 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-21T11:30:18.549 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: libstoragemgmt-1.10.1-1.el9.x86_64 77/96 2026-02-21T11:30:18.549 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/libstoragemgmt.service → /usr/lib/systemd/system/libstoragemgmt.service. 2026-02-21T11:30:18.549 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:18.583 INFO:teuthology.orchestra.run.trial189.stdout: Installing : python3-libstoragemgmt-1.10.1-1.el9.x86_64 78/96 2026-02-21T11:30:19.204 INFO:teuthology.orchestra.run.trial189.stdout: Installing : cryptsetup-2.8.1-3.el9.x86_64 79/96 2026-02-21T11:30:19.966 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-osd-classic-2:20.3.0-5397.gf7273f30.el9.x86_6 80/96 2026-02-21T11:30:20.030 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-osd-crimson-2:20.3.0-5397.gf7273f30.el9.x86_6 81/96 2026-02-21T11:30:20.032 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-volume-2:20.3.0-5397.gf7273f30.el9.noarch 82/96 2026-02-21T11:30:20.047 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-volume-2:20.3.0-5397.gf7273f30.el9.noarch 82/96 2026-02-21T11:30:20.047 INFO:teuthology.orchestra.run.trial189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T11:30:20.047 INFO:teuthology.orchestra.run.trial189.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2026-02-21T11:30:20.047 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:20.057 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-osd-2:20.3.0-5397.gf7273f30.el9.x86_64 83/96 2026-02-21T11:30:20.082 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-osd-2:20.3.0-5397.gf7273f30.el9.x86_64 83/96 2026-02-21T11:30:20.082 INFO:teuthology.orchestra.run.trial189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T11:30:20.082 INFO:teuthology.orchestra.run.trial189.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2026-02-21T11:30:20.082 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-21T11:30:20.082 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2026-02-21T11:30:20.082 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:20.139 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: cephadm-2:20.3.0-5397.gf7273f30.el9.noarch 84/96 2026-02-21T11:30:20.141 INFO:teuthology.orchestra.run.trial189.stdout: Installing : cephadm-2:20.3.0-5397.gf7273f30.el9.noarch 84/96 2026-02-21T11:30:20.150 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-prometheus-alerts-2:20.3.0-5397.gf7273f30.el9 85/96 2026-02-21T11:30:20.174 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-grafana-dashboards-2:20.3.0-5397.gf7273f30.el 86/96 2026-02-21T11:30:20.175 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-mgr-cephadm-2:20.3.0-5397.gf7273f30.el9.noarc 87/96 2026-02-21T11:30:23.078 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-mgr-cephadm-2:20.3.0-5397.gf7273f30.el9.noarc 87/96 2026-02-21T11:30:23.085 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-mgr-dashboard-2:20.3.0-5397.gf7273f30.el9.noa 88/96 2026-02-21T11:30:23.548 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-mgr-dashboard-2:20.3.0-5397.gf7273f30.el9.noa 88/96 2026-02-21T11:30:23.549 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-mgr-diskprediction-local-2:20.3.0-5397.gf7273 89/96 2026-02-21T11:30:23.562 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:20.3.0-5397.gf7273 89/96 2026-02-21T11:30:23.563 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-mgr-k8sevents-2:20.3.0-5397.gf7273f30.el9.noa 90/96 2026-02-21T11:30:23.625 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-mgr-k8sevents-2:20.3.0-5397.gf7273f30.el9.noa 90/96 2026-02-21T11:30:23.667 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-mgr-modules-core-2:20.3.0-5397.gf7273f30.el9. 91/96 2026-02-21T11:30:23.668 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-mgr-2:20.3.0-5397.gf7273f30.el9.x86_64 92/96 2026-02-21T11:30:23.691 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-mgr-2:20.3.0-5397.gf7273f30.el9.x86_64 92/96 2026-02-21T11:30:23.691 INFO:teuthology.orchestra.run.trial189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T11:30:23.691 INFO:teuthology.orchestra.run.trial189.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2026-02-21T11:30:23.692 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-21T11:30:23.692 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2026-02-21T11:30:23.692 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:23.703 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-mgr-rook-2:20.3.0-5397.gf7273f30.el9.noarch 93/96 2026-02-21T11:30:23.923 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-mgr-rook-2:20.3.0-5397.gf7273f30.el9.noarch 93/96 2026-02-21T11:30:23.925 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-mon-2:20.3.0-5397.gf7273f30.el9.x86_64 94/96 2026-02-21T11:30:23.946 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-mon-2:20.3.0-5397.gf7273f30.el9.x86_64 94/96 2026-02-21T11:30:23.946 INFO:teuthology.orchestra.run.trial189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T11:30:23.946 INFO:teuthology.orchestra.run.trial189.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2026-02-21T11:30:23.946 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-21T11:30:23.947 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2026-02-21T11:30:23.947 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:24.042 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-mds-2:20.3.0-5397.gf7273f30.el9.x86_64 95/96 2026-02-21T11:30:24.064 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-mds-2:20.3.0-5397.gf7273f30.el9.x86_64 95/96 2026-02-21T11:30:24.064 INFO:teuthology.orchestra.run.trial189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T11:30:24.064 INFO:teuthology.orchestra.run.trial189.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2026-02-21T11:30:24.064 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-21T11:30:24.064 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2026-02-21T11:30:24.064 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:24.065 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-2:20.3.0-5397.gf7273f30.el9.x86_64 96/96 2026-02-21T11:30:24.073 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-osd-classic-2:20.3.0-5397.gf7273f30.el9.x86_6 96/96 2026-02-21T11:30:24.098 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-osd-crimson-2:20.3.0-5397.gf7273f30.el9.x86_6 96/96 2026-02-21T11:30:24.883 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-2:20.3.0-5397.gf7273f30.el9.x86_64 96/96 2026-02-21T11:30:24.883 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-2:20.3.0-5397.gf7273f30.el9.x86_64 1/96 2026-02-21T11:30:24.883 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-mds-2:20.3.0-5397.gf7273f30.el9.x86_64 2/96 2026-02-21T11:30:24.883 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-mgr-2:20.3.0-5397.gf7273f30.el9.x86_64 3/96 2026-02-21T11:30:24.883 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-mon-2:20.3.0-5397.gf7273f30.el9.x86_64 4/96 2026-02-21T11:30:24.883 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-osd-2:20.3.0-5397.gf7273f30.el9.x86_64 5/96 2026-02-21T11:30:24.883 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-osd-classic-2:20.3.0-5397.gf7273f30.el9.x86_6 6/96 2026-02-21T11:30:24.883 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-osd-crimson-2:20.3.0-5397.gf7273f30.el9.x86_6 7/96 2026-02-21T11:30:24.883 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-grafana-dashboards-2:20.3.0-5397.gf7273f30.el 8/96 2026-02-21T11:30:24.883 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-mgr-cephadm-2:20.3.0-5397.gf7273f30.el9.noarc 9/96 2026-02-21T11:30:24.883 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-mgr-dashboard-2:20.3.0-5397.gf7273f30.el9.noa 10/96 2026-02-21T11:30:24.883 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-mgr-diskprediction-local-2:20.3.0-5397.gf7273 11/96 2026-02-21T11:30:24.883 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-mgr-k8sevents-2:20.3.0-5397.gf7273f30.el9.noa 12/96 2026-02-21T11:30:24.883 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-mgr-modules-core-2:20.3.0-5397.gf7273f30.el9. 13/96 2026-02-21T11:30:24.883 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-mgr-rook-2:20.3.0-5397.gf7273f30.el9.noarch 14/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-prometheus-alerts-2:20.3.0-5397.gf7273f30.el9 15/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-volume-2:20.3.0-5397.gf7273f30.el9.noarch 16/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : cephadm-2:20.3.0-5397.gf7273f30.el9.noarch 17/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : cryptsetup-2.8.1-3.el9.x86_64 18/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ledmon-libs-1.1.0-3.el9.x86_64 19/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libconfig-1.7.2-9.el9.x86_64 20/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libgfortran-11.5.0-14.el9.x86_64 21/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libquadmath-11.5.0-14.el9.x86_64 22/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 23/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-cryptography-36.0.1-5.el9.x86_64 24/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-ply-3.11-14.el9.noarch 25/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 26/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-pyparsing-2.4.7-9.el9.noarch 27/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-requests-2.25.1-10.el9.noarch 28/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-urllib3-1.26.5-7.el9.noarch 29/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : unzip-6.0-59.el9.x86_64 30/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : zip-3.0-35.el9.x86_64 31/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : flexiblas-3.0.4-9.el9.x86_64 32/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : flexiblas-netlib-3.0.4-9.el9.x86_64 33/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 34/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libstoragemgmt-1.10.1-1.el9.x86_64 35/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : lua-5.4.4-4.el9.x86_64 36/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : openblas-0.3.29-1.el9.x86_64 37/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : openblas-openmp-0.3.29-1.el9.x86_64 38/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 39/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-devel-3.9.25-3.el9.x86_64 40/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-jinja2-2.11.3-8.el9.noarch 41/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-jmespath-1.0.1-1.el9.noarch 42/96 2026-02-21T11:30:24.884 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-libstoragemgmt-1.10.1-1.el9.x86_64 43/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-lxml-4.6.5-3.el9.x86_64 44/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 45/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-numpy-1:1.23.5-2.el9.x86_64 46/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-numpy-f2py-1:1.23.5-2.el9.x86_64 47/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 48/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-protobuf-3.14.0-17.el9.noarch 49/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 50/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 51/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 52/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-scipy-1.9.3-2.el9.x86_64 53/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 54/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : xmlsec1-1.2.29-13.el9.x86_64 55/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : xmlsec1-openssl-1.2.29-13.el9.x86_64 56/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 57/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : protobuf-compiler-3.14.0-17.el9.x86_64 58/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : abseil-cpp-20211102.0-4.el9.x86_64 59/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 60/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 61/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 62/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-autocommand-2.2.2-8.el9.noarch 63/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-backports-tarfile-1.2.0-1.el9.noarch 64/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 65/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 66/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 67/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-cheroot-10.0.1-4.el9.noarch 68/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 69/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-google-auth-1:2.45.0-1.el9.noarch 70/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 71/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 72/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-influxdb-5.3.1-1.el9.noarch 73/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-isodate-0.6.1-3.el9.noarch 74/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 75/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 76/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 77/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-jaraco-context-6.0.1-3.el9.noarch 78/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 79/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-jaraco-text-4.0.0-2.el9.noarch 80/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-kubernetes-1:26.1.0-3.el9.noarch 81/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 82/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-msgpack-1.0.3-2.el9.x86_64 83/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 84/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 85/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/96 2026-02-21T11:30:24.885 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 87/96 2026-02-21T11:30:24.886 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 88/96 2026-02-21T11:30:24.886 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 89/96 2026-02-21T11:30:24.886 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-saml-1.16.0-1.el9.noarch 90/96 2026-02-21T11:30:24.886 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 91/96 2026-02-21T11:30:24.886 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-typing-extensions-4.15.0-1.el9.noarch 92/96 2026-02-21T11:30:24.886 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 93/96 2026-02-21T11:30:24.886 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-xmlsec-1.3.13-1.el9.x86_64 94/96 2026-02-21T11:30:24.886 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-xmltodict-0.12.0-15.el9.noarch 95/96 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 96/96 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout:Installed: 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: abseil-cpp-20211102.0-4.el9.x86_64 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-grafana-dashboards-2:20.3.0-5397.gf7273f30.el9.noarch 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mds-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mgr-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mgr-cephadm-2:20.3.0-5397.gf7273f30.el9.noarch 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mgr-dashboard-2:20.3.0-5397.gf7273f30.el9.noarch 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mgr-diskprediction-local-2:20.3.0-5397.gf7273f30.el9.noarch 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mgr-k8sevents-2:20.3.0-5397.gf7273f30.el9.noarch 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mgr-modules-core-2:20.3.0-5397.gf7273f30.el9.noarch 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mgr-rook-2:20.3.0-5397.gf7273f30.el9.noarch 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-mon-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-osd-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-osd-classic-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-osd-crimson-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-prometheus-alerts-2:20.3.0-5397.gf7273f30.el9.noarch 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: ceph-volume-2:20.3.0-5397.gf7273f30.el9.noarch 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: cephadm-2:20.3.0-5397.gf7273f30.el9.noarch 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: cryptsetup-2.8.1-3.el9.x86_64 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: flexiblas-3.0.4-9.el9.x86_64 2026-02-21T11:30:24.952 INFO:teuthology.orchestra.run.trial189.stdout: flexiblas-netlib-3.0.4-9.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: flexiblas-openblas-openmp-3.0.4-9.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: grpc-data-1.46.7-10.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: ledmon-libs-1.1.0-3.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: libconfig-1.7.2-9.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: libgfortran-11.5.0-14.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: libquadmath-11.5.0-14.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: lua-5.4.4-4.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: lua-devel-5.4.4-4.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: luarocks-3.9.2-5.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: openblas-0.3.29-1.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: openblas-openmp-0.3.29-1.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: protobuf-compiler-3.14.0-17.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-autocommand-2.2.2-8.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-babel-2.9.1-2.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-backports-tarfile-1.2.0-1.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-cachetools-4.2.4-1.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-certifi-2023.05.07-4.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-cffi-1.14.5-5.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-cheroot-10.0.1-4.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-cryptography-36.0.1-5.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-devel-3.9.25-3.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-google-auth-1:2.45.0-1.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-influxdb-5.3.1-1.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-isodate-0.6.1-3.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-jaraco-8.2.1-3.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-jaraco-context-6.0.1-3.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2026-02-21T11:30:24.953 INFO:teuthology.orchestra.run.trial189.stdout: python3-jaraco-text-4.0.0-2.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-jinja2-2.11.3-8.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-jmespath-1.0.1-1.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-kubernetes-1:26.1.0-3.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-libstoragemgmt-1.10.1-1.el9.x86_64 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-lxml-4.6.5-3.el9.x86_64 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-msgpack-1.0.3-2.el9.x86_64 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-natsort-7.1.1-5.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-numpy-1:1.23.5-2.el9.x86_64 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-numpy-f2py-1:1.23.5-2.el9.x86_64 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-packaging-20.9-5.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-ply-3.11-14.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-portend-3.1.0-2.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-protobuf-3.14.0-17.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-pycparser-2.20-6.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-pyparsing-2.4.7-9.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-repoze-lru-0.7-16.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-requests-2.25.1-10.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-routes-2.5.1-5.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-rsa-4.9-2.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-saml-1.16.0-1.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-scipy-1.9.3-2.el9.x86_64 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-tempora-5.0.0-2.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-toml-0.10.2-6.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-typing-extensions-4.15.0-1.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-urllib3-1.26.5-7.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-xmlsec-1.3.13-1.el9.x86_64 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-xmltodict-0.12.0-15.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: unzip-6.0-59.el9.x86_64 2026-02-21T11:30:24.954 INFO:teuthology.orchestra.run.trial189.stdout: xmlsec1-1.2.29-13.el9.x86_64 2026-02-21T11:30:24.955 INFO:teuthology.orchestra.run.trial189.stdout: xmlsec1-openssl-1.2.29-13.el9.x86_64 2026-02-21T11:30:24.955 INFO:teuthology.orchestra.run.trial189.stdout: zip-3.0-35.el9.x86_64 2026-02-21T11:30:24.955 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:24.955 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:25.016 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install ceph-base 2026-02-21T11:30:25.322 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:46 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:25.386 INFO:teuthology.orchestra.run.trial189.stdout:Package ceph-base-2:20.3.0-5397.gf7273f30.el9.x86_64 is already installed. 2026-02-21T11:30:25.404 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:25.405 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:25.405 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:25.426 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install cephadm 2026-02-21T11:30:25.737 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:46 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:25.802 INFO:teuthology.orchestra.run.trial189.stdout:Package cephadm-2:20.3.0-5397.gf7273f30.el9.noarch is already installed. 2026-02-21T11:30:25.820 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:25.821 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:25.821 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:25.842 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install ceph-immutable-object-cache 2026-02-21T11:30:26.152 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:47 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:26.235 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:26.235 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:26.235 INFO:teuthology.orchestra.run.trial189.stdout: Package Arch Version Repo Size 2026-02-21T11:30:26.235 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:26.236 INFO:teuthology.orchestra.run.trial189.stdout:Installing: 2026-02-21T11:30:26.236 INFO:teuthology.orchestra.run.trial189.stdout: ceph-immutable-object-cache x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 147 k 2026-02-21T11:30:26.236 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:26.236 INFO:teuthology.orchestra.run.trial189.stdout:Transaction Summary 2026-02-21T11:30:26.236 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:26.236 INFO:teuthology.orchestra.run.trial189.stdout:Install 1 Package 2026-02-21T11:30:26.236 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:26.236 INFO:teuthology.orchestra.run.trial189.stdout:Total download size: 147 k 2026-02-21T11:30:26.236 INFO:teuthology.orchestra.run.trial189.stdout:Installed size: 419 k 2026-02-21T11:30:26.236 INFO:teuthology.orchestra.run.trial189.stdout:Downloading Packages: 2026-02-21T11:30:26.323 INFO:teuthology.orchestra.run.trial189.stdout:ceph-immutable-object-cache-20.3.0-5397.gf7273f 1.7 MB/s | 147 kB 00:00 2026-02-21T11:30:26.323 INFO:teuthology.orchestra.run.trial189.stdout:-------------------------------------------------------------------------------- 2026-02-21T11:30:26.323 INFO:teuthology.orchestra.run.trial189.stdout:Total 1.6 MB/s | 147 kB 00:00 2026-02-21T11:30:26.324 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction check 2026-02-21T11:30:26.327 INFO:teuthology.orchestra.run.trial189.stdout:Transaction check succeeded. 2026-02-21T11:30:26.327 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction test 2026-02-21T11:30:26.361 INFO:teuthology.orchestra.run.trial189.stdout:Transaction test succeeded. 2026-02-21T11:30:26.362 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction 2026-02-21T11:30:26.418 INFO:teuthology.orchestra.run.trial189.stdout: Preparing : 1/1 2026-02-21T11:30:26.419 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-immutable-object-cache-2:20.3.0-5397.gf7273f30. 1/1 2026-02-21T11:30:26.441 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-immutable-object-cache-2:20.3.0-5397.gf7273f30. 1/1 2026-02-21T11:30:26.441 INFO:teuthology.orchestra.run.trial189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T11:30:26.441 INFO:teuthology.orchestra.run.trial189.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2026-02-21T11:30:26.441 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:26.742 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-immutable-object-cache-2:20.3.0-5397.gf7273f30. 1/1 2026-02-21T11:30:26.742 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:26.742 INFO:teuthology.orchestra.run.trial189.stdout:Installed: 2026-02-21T11:30:26.742 INFO:teuthology.orchestra.run.trial189.stdout: ceph-immutable-object-cache-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:26.742 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:26.743 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:26.769 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install ceph-mgr 2026-02-21T11:30:27.076 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:48 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:27.141 INFO:teuthology.orchestra.run.trial189.stdout:Package ceph-mgr-2:20.3.0-5397.gf7273f30.el9.x86_64 is already installed. 2026-02-21T11:30:27.160 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:27.161 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:27.161 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:27.181 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install ceph-mgr-dashboard 2026-02-21T11:30:27.492 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:48 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:27.558 INFO:teuthology.orchestra.run.trial189.stdout:Package ceph-mgr-dashboard-2:20.3.0-5397.gf7273f30.el9.noarch is already installed. 2026-02-21T11:30:27.576 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:27.576 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:27.576 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:27.599 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install ceph-mgr-diskprediction-local 2026-02-21T11:30:27.910 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:48 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:27.973 INFO:teuthology.orchestra.run.trial189.stdout:Package ceph-mgr-diskprediction-local-2:20.3.0-5397.gf7273f30.el9.noarch is already installed. 2026-02-21T11:30:27.992 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:27.992 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:27.992 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:28.013 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install ceph-mgr-rook 2026-02-21T11:30:28.322 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:49 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:28.388 INFO:teuthology.orchestra.run.trial189.stdout:Package ceph-mgr-rook-2:20.3.0-5397.gf7273f30.el9.noarch is already installed. 2026-02-21T11:30:28.406 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:28.407 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:28.407 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:28.430 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install ceph-mgr-cephadm 2026-02-21T11:30:28.736 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:49 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:28.800 INFO:teuthology.orchestra.run.trial189.stdout:Package ceph-mgr-cephadm-2:20.3.0-5397.gf7273f30.el9.noarch is already installed. 2026-02-21T11:30:28.819 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:28.820 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:28.820 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:28.842 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install ceph-osd 2026-02-21T11:30:29.149 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:50 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:29.214 INFO:teuthology.orchestra.run.trial189.stdout:Package ceph-osd-2:20.3.0-5397.gf7273f30.el9.x86_64 is already installed. 2026-02-21T11:30:29.232 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:29.233 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:29.233 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:29.255 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install ceph-osd-classic 2026-02-21T11:30:29.559 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:50 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:29.624 INFO:teuthology.orchestra.run.trial189.stdout:Package ceph-osd-classic-2:20.3.0-5397.gf7273f30.el9.x86_64 is already installed. 2026-02-21T11:30:29.642 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:29.643 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:29.643 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:29.664 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install ceph-fuse 2026-02-21T11:30:29.975 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:50 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:30.058 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:30.058 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:30.058 INFO:teuthology.orchestra.run.trial189.stdout: Package Arch Version Repo Size 2026-02-21T11:30:30.059 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:30.059 INFO:teuthology.orchestra.run.trial189.stdout:Installing: 2026-02-21T11:30:30.059 INFO:teuthology.orchestra.run.trial189.stdout: ceph-fuse x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 988 k 2026-02-21T11:30:30.059 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:30.059 INFO:teuthology.orchestra.run.trial189.stdout:Transaction Summary 2026-02-21T11:30:30.059 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:30.059 INFO:teuthology.orchestra.run.trial189.stdout:Install 1 Package 2026-02-21T11:30:30.059 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:30.059 INFO:teuthology.orchestra.run.trial189.stdout:Total download size: 988 k 2026-02-21T11:30:30.059 INFO:teuthology.orchestra.run.trial189.stdout:Installed size: 2.9 M 2026-02-21T11:30:30.059 INFO:teuthology.orchestra.run.trial189.stdout:Downloading Packages: 2026-02-21T11:30:30.626 INFO:teuthology.orchestra.run.trial189.stdout:ceph-fuse-20.3.0-5397.gf7273f30.el9.x86_64.rpm 1.7 MB/s | 988 kB 00:00 2026-02-21T11:30:30.627 INFO:teuthology.orchestra.run.trial189.stdout:-------------------------------------------------------------------------------- 2026-02-21T11:30:30.627 INFO:teuthology.orchestra.run.trial189.stdout:Total 1.7 MB/s | 988 kB 00:00 2026-02-21T11:30:30.627 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction check 2026-02-21T11:30:30.632 INFO:teuthology.orchestra.run.trial189.stdout:Transaction check succeeded. 2026-02-21T11:30:30.632 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction test 2026-02-21T11:30:30.667 INFO:teuthology.orchestra.run.trial189.stdout:Transaction test succeeded. 2026-02-21T11:30:30.667 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction 2026-02-21T11:30:30.759 INFO:teuthology.orchestra.run.trial189.stdout: Preparing : 1/1 2026-02-21T11:30:30.776 INFO:teuthology.orchestra.run.trial189.stdout: Installing : ceph-fuse-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:31.008 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: ceph-fuse-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:31.064 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : ceph-fuse-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:31.064 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:31.065 INFO:teuthology.orchestra.run.trial189.stdout:Installed: 2026-02-21T11:30:31.065 INFO:teuthology.orchestra.run.trial189.stdout: ceph-fuse-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:31.065 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:31.065 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:31.092 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install ceph-volume 2026-02-21T11:30:31.404 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:52 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:31.468 INFO:teuthology.orchestra.run.trial189.stdout:Package ceph-volume-2:20.3.0-5397.gf7273f30.el9.noarch is already installed. 2026-02-21T11:30:31.486 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:31.487 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:31.487 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:31.508 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install librados-devel 2026-02-21T11:30:31.814 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:52 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:31.898 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:31.899 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:31.899 INFO:teuthology.orchestra.run.trial189.stdout: Package Arch Version Repo Size 2026-02-21T11:30:31.899 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:31.899 INFO:teuthology.orchestra.run.trial189.stdout:Installing: 2026-02-21T11:30:31.899 INFO:teuthology.orchestra.run.trial189.stdout: librados-devel x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 123 k 2026-02-21T11:30:31.899 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:31.899 INFO:teuthology.orchestra.run.trial189.stdout:Transaction Summary 2026-02-21T11:30:31.899 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:31.899 INFO:teuthology.orchestra.run.trial189.stdout:Install 1 Package 2026-02-21T11:30:31.900 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:31.900 INFO:teuthology.orchestra.run.trial189.stdout:Total download size: 123 k 2026-02-21T11:30:31.900 INFO:teuthology.orchestra.run.trial189.stdout:Installed size: 437 k 2026-02-21T11:30:31.900 INFO:teuthology.orchestra.run.trial189.stdout:Downloading Packages: 2026-02-21T11:30:31.986 INFO:teuthology.orchestra.run.trial189.stdout:librados-devel-20.3.0-5397.gf7273f30.el9.x86_64 1.4 MB/s | 123 kB 00:00 2026-02-21T11:30:31.987 INFO:teuthology.orchestra.run.trial189.stdout:-------------------------------------------------------------------------------- 2026-02-21T11:30:31.987 INFO:teuthology.orchestra.run.trial189.stdout:Total 1.4 MB/s | 123 kB 00:00 2026-02-21T11:30:31.987 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction check 2026-02-21T11:30:31.989 INFO:teuthology.orchestra.run.trial189.stdout:Transaction check succeeded. 2026-02-21T11:30:31.990 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction test 2026-02-21T11:30:32.021 INFO:teuthology.orchestra.run.trial189.stdout:Transaction test succeeded. 2026-02-21T11:30:32.021 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction 2026-02-21T11:30:32.075 INFO:teuthology.orchestra.run.trial189.stdout: Preparing : 1/1 2026-02-21T11:30:32.091 INFO:teuthology.orchestra.run.trial189.stdout: Installing : librados-devel-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:32.171 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: librados-devel-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:32.229 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : librados-devel-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:32.229 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:32.230 INFO:teuthology.orchestra.run.trial189.stdout:Installed: 2026-02-21T11:30:32.230 INFO:teuthology.orchestra.run.trial189.stdout: librados-devel-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:32.230 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:32.230 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:32.256 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install libcephfs2 2026-02-21T11:30:32.568 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:53 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:32.632 INFO:teuthology.orchestra.run.trial189.stdout:Package libcephfs2-2:20.3.0-5397.gf7273f30.el9.x86_64 is already installed. 2026-02-21T11:30:32.650 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:32.650 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:32.650 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:32.672 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install libcephfs-devel 2026-02-21T11:30:32.977 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:53 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:33.060 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:33.060 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:33.061 INFO:teuthology.orchestra.run.trial189.stdout: Package Arch Version Repo Size 2026-02-21T11:30:33.061 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:33.061 INFO:teuthology.orchestra.run.trial189.stdout:Installing: 2026-02-21T11:30:33.061 INFO:teuthology.orchestra.run.trial189.stdout: libcephfs-devel x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 37 k 2026-02-21T11:30:33.061 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:33.061 INFO:teuthology.orchestra.run.trial189.stdout:Transaction Summary 2026-02-21T11:30:33.061 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:33.061 INFO:teuthology.orchestra.run.trial189.stdout:Install 1 Package 2026-02-21T11:30:33.061 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:33.061 INFO:teuthology.orchestra.run.trial189.stdout:Total download size: 37 k 2026-02-21T11:30:33.061 INFO:teuthology.orchestra.run.trial189.stdout:Installed size: 171 k 2026-02-21T11:30:33.061 INFO:teuthology.orchestra.run.trial189.stdout:Downloading Packages: 2026-02-21T11:30:33.124 INFO:teuthology.orchestra.run.trial189.stdout:libcephfs-devel-20.3.0-5397.gf7273f30.el9.x86_6 592 kB/s | 37 kB 00:00 2026-02-21T11:30:33.125 INFO:teuthology.orchestra.run.trial189.stdout:-------------------------------------------------------------------------------- 2026-02-21T11:30:33.125 INFO:teuthology.orchestra.run.trial189.stdout:Total 581 kB/s | 37 kB 00:00 2026-02-21T11:30:33.125 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction check 2026-02-21T11:30:33.128 INFO:teuthology.orchestra.run.trial189.stdout:Transaction check succeeded. 2026-02-21T11:30:33.128 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction test 2026-02-21T11:30:33.130 INFO:teuthology.orchestra.run.trial189.stdout:Transaction test succeeded. 2026-02-21T11:30:33.130 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction 2026-02-21T11:30:33.151 INFO:teuthology.orchestra.run.trial189.stdout: Preparing : 1/1 2026-02-21T11:30:33.168 INFO:teuthology.orchestra.run.trial189.stdout: Installing : libcephfs-devel-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:33.224 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: libcephfs-devel-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:33.281 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : libcephfs-devel-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:33.281 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:33.281 INFO:teuthology.orchestra.run.trial189.stdout:Installed: 2026-02-21T11:30:33.281 INFO:teuthology.orchestra.run.trial189.stdout: libcephfs-devel-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:33.281 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:33.282 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:33.309 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install librados2 2026-02-21T11:30:33.617 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:54 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:33.681 INFO:teuthology.orchestra.run.trial189.stdout:Package librados2-2:20.3.0-5397.gf7273f30.el9.x86_64 is already installed. 2026-02-21T11:30:33.700 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:33.700 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:33.701 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:33.722 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install librbd1 2026-02-21T11:30:34.030 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:55 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:34.094 INFO:teuthology.orchestra.run.trial189.stdout:Package librbd1-2:20.3.0-5397.gf7273f30.el9.x86_64 is already installed. 2026-02-21T11:30:34.112 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:34.113 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:34.113 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:34.135 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install python3-rados 2026-02-21T11:30:34.444 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:55 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:34.509 INFO:teuthology.orchestra.run.trial189.stdout:Package python3-rados-2:20.3.0-5397.gf7273f30.el9.x86_64 is already installed. 2026-02-21T11:30:34.527 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:34.527 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:34.527 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:34.548 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install python3-rgw 2026-02-21T11:30:34.871 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:55 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:34.936 INFO:teuthology.orchestra.run.trial189.stdout:Package python3-rgw-2:20.3.0-5397.gf7273f30.el9.x86_64 is already installed. 2026-02-21T11:30:34.954 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:34.954 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:34.954 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:34.977 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install python3-cephfs 2026-02-21T11:30:35.285 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:35.350 INFO:teuthology.orchestra.run.trial189.stdout:Package python3-cephfs-2:20.3.0-5397.gf7273f30.el9.x86_64 is already installed. 2026-02-21T11:30:35.368 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:35.368 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:35.368 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:35.390 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install python3-rbd 2026-02-21T11:30:35.696 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:35.760 INFO:teuthology.orchestra.run.trial189.stdout:Package python3-rbd-2:20.3.0-5397.gf7273f30.el9.x86_64 is already installed. 2026-02-21T11:30:35.779 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:35.780 INFO:teuthology.orchestra.run.trial189.stdout:Nothing to do. 2026-02-21T11:30:35.780 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:35.800 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install rbd-fuse 2026-02-21T11:30:36.107 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:57 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:36.191 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:36.192 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:36.192 INFO:teuthology.orchestra.run.trial189.stdout: Package Architecture Version Repository Size 2026-02-21T11:30:36.192 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:36.192 INFO:teuthology.orchestra.run.trial189.stdout:Installing: 2026-02-21T11:30:36.192 INFO:teuthology.orchestra.run.trial189.stdout: rbd-fuse x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 84 k 2026-02-21T11:30:36.192 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:36.192 INFO:teuthology.orchestra.run.trial189.stdout:Transaction Summary 2026-02-21T11:30:36.192 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:36.192 INFO:teuthology.orchestra.run.trial189.stdout:Install 1 Package 2026-02-21T11:30:36.192 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:36.192 INFO:teuthology.orchestra.run.trial189.stdout:Total download size: 84 k 2026-02-21T11:30:36.192 INFO:teuthology.orchestra.run.trial189.stdout:Installed size: 209 k 2026-02-21T11:30:36.192 INFO:teuthology.orchestra.run.trial189.stdout:Downloading Packages: 2026-02-21T11:30:36.552 INFO:teuthology.orchestra.run.trial189.stdout:rbd-fuse-20.3.0-5397.gf7273f30.el9.x86_64.rpm 235 kB/s | 84 kB 00:00 2026-02-21T11:30:36.552 INFO:teuthology.orchestra.run.trial189.stdout:-------------------------------------------------------------------------------- 2026-02-21T11:30:36.553 INFO:teuthology.orchestra.run.trial189.stdout:Total 234 kB/s | 84 kB 00:00 2026-02-21T11:30:36.553 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction check 2026-02-21T11:30:36.556 INFO:teuthology.orchestra.run.trial189.stdout:Transaction check succeeded. 2026-02-21T11:30:36.556 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction test 2026-02-21T11:30:36.587 INFO:teuthology.orchestra.run.trial189.stdout:Transaction test succeeded. 2026-02-21T11:30:36.587 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction 2026-02-21T11:30:36.639 INFO:teuthology.orchestra.run.trial189.stdout: Preparing : 1/1 2026-02-21T11:30:36.656 INFO:teuthology.orchestra.run.trial189.stdout: Installing : rbd-fuse-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:36.738 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: rbd-fuse-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:36.796 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : rbd-fuse-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:36.796 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:36.796 INFO:teuthology.orchestra.run.trial189.stdout:Installed: 2026-02-21T11:30:36.796 INFO:teuthology.orchestra.run.trial189.stdout: rbd-fuse-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:36.796 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:36.796 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:36.825 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install rbd-mirror 2026-02-21T11:30:37.138 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:00:58 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:37.221 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:37.221 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:37.221 INFO:teuthology.orchestra.run.trial189.stdout: Package Arch Version Repo Size 2026-02-21T11:30:37.221 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:37.221 INFO:teuthology.orchestra.run.trial189.stdout:Installing: 2026-02-21T11:30:37.222 INFO:teuthology.orchestra.run.trial189.stdout: rbd-mirror x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 2.9 M 2026-02-21T11:30:37.222 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:37.222 INFO:teuthology.orchestra.run.trial189.stdout:Transaction Summary 2026-02-21T11:30:37.222 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:37.222 INFO:teuthology.orchestra.run.trial189.stdout:Install 1 Package 2026-02-21T11:30:37.222 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:37.222 INFO:teuthology.orchestra.run.trial189.stdout:Total download size: 2.9 M 2026-02-21T11:30:37.222 INFO:teuthology.orchestra.run.trial189.stdout:Installed size: 11 M 2026-02-21T11:30:37.222 INFO:teuthology.orchestra.run.trial189.stdout:Downloading Packages: 2026-02-21T11:30:38.724 INFO:teuthology.orchestra.run.trial189.stdout:rbd-mirror-20.3.0-5397.gf7273f30.el9.x86_64.rpm 1.9 MB/s | 2.9 MB 00:01 2026-02-21T11:30:38.724 INFO:teuthology.orchestra.run.trial189.stdout:-------------------------------------------------------------------------------- 2026-02-21T11:30:38.724 INFO:teuthology.orchestra.run.trial189.stdout:Total 1.9 MB/s | 2.9 MB 00:01 2026-02-21T11:30:38.724 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction check 2026-02-21T11:30:38.729 INFO:teuthology.orchestra.run.trial189.stdout:Transaction check succeeded. 2026-02-21T11:30:38.729 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction test 2026-02-21T11:30:38.768 INFO:teuthology.orchestra.run.trial189.stdout:Transaction test succeeded. 2026-02-21T11:30:38.768 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction 2026-02-21T11:30:38.946 INFO:teuthology.orchestra.run.trial189.stdout: Preparing : 1/1 2026-02-21T11:30:38.947 INFO:teuthology.orchestra.run.trial189.stdout: Installing : rbd-mirror-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:38.970 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: rbd-mirror-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:38.970 INFO:teuthology.orchestra.run.trial189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2026-02-21T11:30:38.970 INFO:teuthology.orchestra.run.trial189.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2026-02-21T11:30:38.970 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-21T11:30:38.970 INFO:teuthology.orchestra.run.trial189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2026-02-21T11:30:38.970 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:39.296 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : rbd-mirror-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:39.296 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:39.296 INFO:teuthology.orchestra.run.trial189.stdout:Installed: 2026-02-21T11:30:39.296 INFO:teuthology.orchestra.run.trial189.stdout: rbd-mirror-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:39.296 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:39.297 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:39.327 DEBUG:teuthology.orchestra.run.trial189:> sudo yum -y install rbd-nbd 2026-02-21T11:30:39.637 INFO:teuthology.orchestra.run.trial189.stdout:Last metadata expiration check: 0:01:00 ago on Sat 21 Feb 2026 11:29:39 AM UTC. 2026-02-21T11:30:39.721 INFO:teuthology.orchestra.run.trial189.stdout:Dependencies resolved. 2026-02-21T11:30:39.721 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:39.721 INFO:teuthology.orchestra.run.trial189.stdout: Package Architecture Version Repository Size 2026-02-21T11:30:39.721 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:39.721 INFO:teuthology.orchestra.run.trial189.stdout:Installing: 2026-02-21T11:30:39.721 INFO:teuthology.orchestra.run.trial189.stdout: rbd-nbd x86_64 2:20.3.0-5397.gf7273f30.el9 ceph 172 k 2026-02-21T11:30:39.721 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:39.721 INFO:teuthology.orchestra.run.trial189.stdout:Transaction Summary 2026-02-21T11:30:39.721 INFO:teuthology.orchestra.run.trial189.stdout:================================================================================ 2026-02-21T11:30:39.721 INFO:teuthology.orchestra.run.trial189.stdout:Install 1 Package 2026-02-21T11:30:39.721 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:39.721 INFO:teuthology.orchestra.run.trial189.stdout:Total download size: 172 k 2026-02-21T11:30:39.721 INFO:teuthology.orchestra.run.trial189.stdout:Installed size: 462 k 2026-02-21T11:30:39.721 INFO:teuthology.orchestra.run.trial189.stdout:Downloading Packages: 2026-02-21T11:30:39.809 INFO:teuthology.orchestra.run.trial189.stdout:rbd-nbd-20.3.0-5397.gf7273f30.el9.x86_64.rpm 1.9 MB/s | 172 kB 00:00 2026-02-21T11:30:39.809 INFO:teuthology.orchestra.run.trial189.stdout:-------------------------------------------------------------------------------- 2026-02-21T11:30:39.810 INFO:teuthology.orchestra.run.trial189.stdout:Total 1.9 MB/s | 172 kB 00:00 2026-02-21T11:30:39.810 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction check 2026-02-21T11:30:39.813 INFO:teuthology.orchestra.run.trial189.stdout:Transaction check succeeded. 2026-02-21T11:30:39.813 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction test 2026-02-21T11:30:39.845 INFO:teuthology.orchestra.run.trial189.stdout:Transaction test succeeded. 2026-02-21T11:30:39.845 INFO:teuthology.orchestra.run.trial189.stdout:Running transaction 2026-02-21T11:30:39.902 INFO:teuthology.orchestra.run.trial189.stdout: Preparing : 1/1 2026-02-21T11:30:39.918 INFO:teuthology.orchestra.run.trial189.stdout: Installing : rbd-nbd-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:40.000 INFO:teuthology.orchestra.run.trial189.stdout: Running scriptlet: rbd-nbd-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:40.058 INFO:teuthology.orchestra.run.trial189.stdout: Verifying : rbd-nbd-2:20.3.0-5397.gf7273f30.el9.x86_64 1/1 2026-02-21T11:30:40.058 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:40.058 INFO:teuthology.orchestra.run.trial189.stdout:Installed: 2026-02-21T11:30:40.058 INFO:teuthology.orchestra.run.trial189.stdout: rbd-nbd-2:20.3.0-5397.gf7273f30.el9.x86_64 2026-02-21T11:30:40.058 INFO:teuthology.orchestra.run.trial189.stdout: 2026-02-21T11:30:40.058 INFO:teuthology.orchestra.run.trial189.stdout:Complete! 2026-02-21T11:30:40.090 DEBUG:teuthology.parallel:result is None 2026-02-21T11:30:40.091 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f7273f3049146bdd7fa7b4a400114872a50228e9 2026-02-21T11:30:40.257 DEBUG:teuthology.orchestra.run.trial189:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2026-02-21T11:30:40.278 INFO:teuthology.orchestra.run.trial189.stdout:20.3.0-5397.gf7273f30.el9 2026-02-21T11:30:40.278 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5397.gf7273f30.el9 2026-02-21T11:30:40.278 INFO:teuthology.task.install:The correct ceph version 20.3.0-5397.gf7273f30 is installed. 2026-02-21T11:30:40.280 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-21T11:30:40.281 DEBUG:teuthology.orchestra.run.trial189:> set -ex 2026-02-21T11:30:40.281 DEBUG:teuthology.orchestra.run.trial189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-21T11:30:40.342 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-21T11:30:40.343 DEBUG:teuthology.orchestra.run.trial189:> set -ex 2026-02-21T11:30:40.343 DEBUG:teuthology.orchestra.run.trial189:> sudo dd of=/usr/bin/daemon-helper 2026-02-21T11:30:40.407 DEBUG:teuthology.orchestra.run.trial189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-21T11:30:40.471 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-21T11:30:40.471 DEBUG:teuthology.orchestra.run.trial189:> set -ex 2026-02-21T11:30:40.471 DEBUG:teuthology.orchestra.run.trial189:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-21T11:30:40.535 DEBUG:teuthology.orchestra.run.trial189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-21T11:30:40.598 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-21T11:30:40.598 DEBUG:teuthology.orchestra.run.trial189:> set -ex 2026-02-21T11:30:40.598 DEBUG:teuthology.orchestra.run.trial189:> sudo dd of=/usr/bin/stdin-killer 2026-02-21T11:30:40.662 DEBUG:teuthology.orchestra.run.trial189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-21T11:30:40.726 INFO:teuthology.run_tasks:Running task workunit... 2026-02-21T11:30:40.745 INFO:tasks.workunit:Pulling workunits from ref f7273f3049146bdd7fa7b4a400114872a50228e9 2026-02-21T11:30:40.745 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-21T11:30:40.745 INFO:tasks.workunit:timeout=3h 2026-02-21T11:30:40.745 INFO:tasks.workunit:cleanup=True 2026-02-21T11:30:40.746 DEBUG:teuthology.orchestra.run.trial189:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-21T11:30:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T11:30:40.784 INFO:teuthology.orchestra.run.trial189.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-21T11:30:40.785 DEBUG:teuthology.orchestra.run.trial189:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-21T11:30:40.840 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-21T11:30:40.840 DEBUG:teuthology.orchestra.run.trial189:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-21T11:30:40.896 DEBUG:teuthology.orchestra.run.trial189:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout f7273f3049146bdd7fa7b4a400114872a50228e9 2026-02-21T11:30:40.959 INFO:tasks.workunit.client.0.trial189.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-21T11:31:00.432 INFO:tasks.workunit.client.0.trial189.stderr:Note: switching to 'f7273f3049146bdd7fa7b4a400114872a50228e9'. 2026-02-21T11:31:00.432 INFO:tasks.workunit.client.0.trial189.stderr: 2026-02-21T11:31:00.432 INFO:tasks.workunit.client.0.trial189.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-21T11:31:00.432 INFO:tasks.workunit.client.0.trial189.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-21T11:31:00.432 INFO:tasks.workunit.client.0.trial189.stderr:state without impacting any branches by switching back to a branch. 2026-02-21T11:31:00.432 INFO:tasks.workunit.client.0.trial189.stderr: 2026-02-21T11:31:00.432 INFO:tasks.workunit.client.0.trial189.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-21T11:31:00.432 INFO:tasks.workunit.client.0.trial189.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-21T11:31:00.432 INFO:tasks.workunit.client.0.trial189.stderr: 2026-02-21T11:31:00.432 INFO:tasks.workunit.client.0.trial189.stderr: git switch -c 2026-02-21T11:31:00.432 INFO:tasks.workunit.client.0.trial189.stderr: 2026-02-21T11:31:00.433 INFO:tasks.workunit.client.0.trial189.stderr:Or undo this operation with: 2026-02-21T11:31:00.433 INFO:tasks.workunit.client.0.trial189.stderr: 2026-02-21T11:31:00.433 INFO:tasks.workunit.client.0.trial189.stderr: git switch - 2026-02-21T11:31:00.433 INFO:tasks.workunit.client.0.trial189.stderr: 2026-02-21T11:31:00.433 INFO:tasks.workunit.client.0.trial189.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-21T11:31:00.433 INFO:tasks.workunit.client.0.trial189.stderr: 2026-02-21T11:31:00.433 INFO:tasks.workunit.client.0.trial189.stderr:HEAD is now at f7273f30491 Merge branch 'fs_mirror_daemon_status' of https://github.com/kotreshhr/ceph into wip-bharath21-testing-2026-02-20-1039 2026-02-21T11:31:00.439 DEBUG:teuthology.orchestra.run.trial189:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-21T11:31:00.500 DEBUG:teuthology.orchestra.run.trial189:> set -ex 2026-02-21T11:31:00.500 DEBUG:teuthology.orchestra.run.trial189:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-21T11:31:00.560 INFO:tasks.workunit:Running workunits matching misc on client.0... 2026-02-21T11:31:00.561 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2026-02-21T11:31:00.561 DEBUG:teuthology.orchestra.run.trial189:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2026-02-21T11:31:00.623 INFO:tasks.workunit.client.0.trial189.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-21T11:31:00.628 INFO:tasks.workunit.client.0.trial189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-21T11:31:00.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-21T11:31:00.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-21T11:31:00.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: main: export PYTHONWARNINGS=ignore 2026-02-21T11:31:00.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: main: PYTHONWARNINGS=ignore 2026-02-21T11:31:00.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: main: export CEPH_CONF=/dev/null 2026-02-21T11:31:00.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: main: CEPH_CONF=/dev/null 2026-02-21T11:31:00.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: main: unset CEPH_ARGS 2026-02-21T11:31:00.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: main: local code 2026-02-21T11:31:00.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: main: run td/mclock-config 2026-02-21T11:31:00.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2026-02-21T11:31:00.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2026-02-21T11:31:00.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2026-02-21T11:31:00.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2026-02-21T11:31:00.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2026-02-21T11:31:00.629 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2026-02-21T11:31:00.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=e6bec3cc-fd12-4810-a327-e5c32c674d23 --auth-supported=none ' 2026-02-21T11:31:00.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2026-02-21T11:31:00.630 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2026-02-21T11:31:00.632 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2026-02-21T11:31:00.632 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-21T11:31:00.632 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2026-02-21T11:31:00.632 INFO:tasks.workunit.client.0.trial189.stderr:TEST_profile_builtin_to_custom 2026-02-21T11:31:00.632 INFO:tasks.workunit.client.0.trial189.stderr:TEST_profile_custom_to_builtin 2026-02-21T11:31:00.633 INFO:tasks.workunit.client.0.trial189.stderr:TEST_recovery_limit_adjustment_mclock' 2026-02-21T11:31:00.633 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-21T11:31:00.633 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-21T11:31:00.633 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T11:31:00.633 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T11:31:00.633 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:00.633 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:00.633 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:00.633 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:00.633 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:00.634 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:00.634 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:00.634 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:00.634 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:00.634 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:00.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:00.636 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:00.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:00.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:00.636 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:00.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:00.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:00.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:00.636 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:00.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:00.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:00.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:00.638 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:00.638 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:00.638 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:00.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:00.640 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:00.640 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:00.640 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T11:31:00.640 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:31:00.640 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:00.640 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:00.640 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.37640 2026-02-21T11:31:00.642 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:31:00.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-21T11:31:00.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-21T11:31:00.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:31:00.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T11:31:00.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2026-02-21T11:31:00.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2026-02-21T11:31:00.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2026-02-21T11:31:00.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T11:31:00.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T11:31:00.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:00.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:00.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:00.644 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:00.644 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:00.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:00.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:00.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:00.645 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:00.646 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:00.646 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:00.646 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:00.646 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:00.646 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:00.646 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:00.647 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:00.647 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:00.647 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:00.647 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:00.648 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:00.648 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:00.648 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:00.649 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:00.650 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:00.650 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:00.650 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:00.650 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:00.651 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:00.651 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T11:31:00.651 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:31:00.652 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:00.652 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:00.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.37640 2026-02-21T11:31:00.654 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:31:00.654 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:31:00.654 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:31:00.654 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T11:31:00.654 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2026-02-21T11:31:00.654 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-21T11:31:00.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:31:00.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:31:00.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:31:00.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-21T11:31:00.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-21T11:31:00.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:31:00.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:00.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:00.673 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:00.673 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:00.673 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:00.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:00.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T11:31:00.696 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:31:00.697 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:31:00.697 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:31:00.697 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:31:00.697 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:31:00.698 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:31:00.698 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:31:00.698 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:31:00.698 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:31:00.699 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:00.699 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:00.699 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:00.699 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-mon.a.asok 2026-02-21T11:31:00.699 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:31:00.699 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.37640/ceph-mon.a.asok config get fsid 2026-02-21T11:31:00.740 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:31:00.740 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:31:00.740 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:31:00.740 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:31:00.740 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:31:00.740 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:31:00.741 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:31:00.741 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:31:00.741 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:00.741 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:00.741 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:00.741 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-mon.a.asok 2026-02-21T11:31:00.741 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:31:00.741 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.37640/ceph-mon.a.asok config get mon_host 2026-02-21T11:31:00.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2026-02-21T11:31:00.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-21T11:31:00.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:31:00.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:31:00.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:31:00.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-21T11:31:00.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:31:00.882 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:31:00.882 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:00.882 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:00.882 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:00.883 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:00.883 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:00.883 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:00.883 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:31:00.883 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:31:00.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-21T11:31:00.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-21T11:31:00.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:31:00.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e6bec3cc-fd12-4810-a327-e5c32c674d23 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:00.900 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:00.901 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:00.901 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:00.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:00.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:31:00.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:31:00.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:31:00.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-21T11:31:00.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-21T11:31:00.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:31:00.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:31:00.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:31:00.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:31:00.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:31:00.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-21T11:31:00.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-21T11:31:00.902 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:31:00.902 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=840ead89-d67b-4eea-9467-9e25f176bdf6 2026-02-21T11:31:00.902 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 840ead89-d67b-4eea-9467-9e25f176bdf6' 2026-02-21T11:31:00.902 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 840ead89-d67b-4eea-9467-9e25f176bdf6 2026-02-21T11:31:00.902 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:31:00.909 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB0l5lp9ngeNhAAGL4QlKnh+l4FKXjlW8zvow== 2026-02-21T11:31:00.909 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB0l5lp9ngeNhAAGL4QlKnh+l4FKXjlW8zvow=="}' 2026-02-21T11:31:00.909 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 840ead89-d67b-4eea-9467-9e25f176bdf6 -i td/mclock-config/0/new.json 2026-02-21T11:31:01.012 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:31:01.019 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-21T11:31:01.020 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e6bec3cc-fd12-4810-a327-e5c32c674d23 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQB0l5lp9ngeNhAAGL4QlKnh+l4FKXjlW8zvow== --osd-uuid 840ead89-d67b-4eea-9467-9e25f176bdf6 2026-02-21T11:31:01.035 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:01.032+0000 7f8bf6240780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:01.036 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:01.034+0000 7f8bf6240780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:01.036 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:01.035+0000 7f8bf6240780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:01.037 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:01.035+0000 7f8bf6240780 -1 bdev(0x2444c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:31:01.037 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:01.035+0000 7f8bf6240780 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-21T11:31:01.472 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-21T11:31:01.473 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:31:01.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:31:01.474 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:31:01.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:31:01.574 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:31:01.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:31:01.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e6bec3cc-fd12-4810-a327-e5c32c674d23 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-21T11:31:01.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:31:01.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:31:01.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:31:01.588 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:01.586+0000 7f191d842780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:01.589 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:01.587+0000 7f191d842780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:01.590 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:01.588+0000 7f191d842780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:01.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:31:01.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:31:01.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:31:01.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:31:01.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:31:01.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:01.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:31:01.670 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:31:01.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:01.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:31:01.765 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:31:01.798 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:01.795+0000 7f191d842780 -1 Falling back to public interface 2026-02-21T11:31:02.095 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:02.092+0000 7f191d842780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:31:02.766 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:31:02.766 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:02.766 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:31:02.767 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:31:02.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:02.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:31:02.863 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/162483504,v1:127.0.0.1:6801/162483504] [v2:127.0.0.1:6802/162483504,v1:127.0.0.1:6803/162483504] exists,up 840ead89-d67b-4eea-9467-9e25f176bdf6 2026-02-21T11:31:02.865 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:31:02.865 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:31:02.865 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:31:02.865 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-21T11:31:02.865 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T11:31:02.865 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T11:31:02.865 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:02.866 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:02.866 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:02.866 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:02.866 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:02.866 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:02.866 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T11:31:02.866 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T11:31:02.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2026-02-21T11:31:02.913 INFO:tasks.workunit.client.0.trial189.stdout:osd_max_backfills: 1 2026-02-21T11:31:02.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2026-02-21T11:31:02.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2026-02-21T11:31:03.012 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2026-02-21T11:31:05.014 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-21T11:31:05.014 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T11:31:05.015 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T11:31:05.015 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:05.015 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:05.015 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:05.015 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:05.016 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:05.016 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:05.016 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T11:31:05.016 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T11:31:05.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2026-02-21T11:31:05.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2026-02-21T11:31:05.064 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2026-02-21T11:31:05.064 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T11:31:05.064 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T11:31:05.064 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:05.064 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:05.064 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:05.064 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:05.064 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:05.065 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:05.065 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T11:31:05.065 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok dump_recovery_reservations 2026-02-21T11:31:05.110 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2026-02-21T11:31:05.111 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2026-02-21T11:31:05.112 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2026-02-21T11:31:05.112 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T11:31:05.113 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T11:31:05.113 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:05.113 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:05.113 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:05.113 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:05.113 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:05.113 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:05.113 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T11:31:05.113 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok dump_recovery_reservations 2026-02-21T11:31:05.158 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2026-02-21T11:31:05.158 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2026-02-21T11:31:05.158 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2026-02-21T11:31:05.386 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2026-02-21T11:31:05.610 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2026-02-21T11:31:07.612 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-21T11:31:07.613 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T11:31:07.613 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T11:31:07.613 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:07.613 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:07.614 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:07.614 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:07.614 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:07.615 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:07.615 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T11:31:07.615 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T11:31:07.660 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-21T11:31:07.660 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-21T11:31:07.662 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2026-02-21T11:31:07.662 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T11:31:07.662 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T11:31:07.662 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:07.662 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:07.662 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:07.662 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:07.662 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:07.662 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:07.662 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T11:31:07.663 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok dump_recovery_reservations 2026-02-21T11:31:07.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-21T11:31:07.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-21T11:31:07.707 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2026-02-21T11:31:07.707 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T11:31:07.709 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T11:31:07.709 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:07.709 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:07.709 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:07.709 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:07.709 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:07.709 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:07.709 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T11:31:07.710 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok dump_recovery_reservations 2026-02-21T11:31:07.755 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-21T11:31:07.755 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-21T11:31:07.755 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2026-02-21T11:31:07.755 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:07.755 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:07.756 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:07.756 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:07.756 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:07.860 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:07.860 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2026-02-21T11:31:08.082 INFO:tasks.workunit.client.0.trial189.stderr:osd.0 is already down. 2026-02-21T11:31:08.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2026-02-21T11:31:08.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2026-02-21T11:31:08.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:31:08.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:31:08.090 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:31:08.090 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:08.090 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:31:08.090 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:31:08.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:08.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2026-02-21T11:31:08.310 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6800/162483504,v1:127.0.0.1:6801/162483504] [v2:127.0.0.1:6802/162483504,v1:127.0.0.1:6803/162483504] exists 840ead89-d67b-4eea-9467-9e25f176bdf6 2026-02-21T11:31:08.314 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:31:08.314 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:31:08.314 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:31:08.314 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2026-02-21T11:31:08.314 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/mclock-config 2026-02-21T11:31:08.314 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T11:31:08.314 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-21T11:31:08.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T11:31:08.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/mclock-config/0 2026-02-21T11:31:08.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=e6bec3cc-fd12-4810-a327-e5c32c674d23 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-21T11:31:08.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:31:08.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:31:08.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:31:08.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-21T11:31:08.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-21T11:31:08.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T11:31:08.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T11:31:08.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-21T11:31:08.316 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T11:31:08.316 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:08.316 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:08.316 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:08.316 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:08.316 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:08.316 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:08.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:08.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:31:08.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-21T11:31:08.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-21T11:31:08.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:31:08.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:31:08.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:31:08.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:31:08.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T11:31:08.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2026-02-21T11:31:08.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/mclock-config/0 2026-02-21T11:31:08.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-21T11:31:08.317 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:31:08.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=e6bec3cc-fd12-4810-a327-e5c32c674d23 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2026-02-21T11:31:08.318 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/mclock-config/0/whoami 2026-02-21T11:31:08.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-21T11:31:08.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T11:31:08.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T11:31:08.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T11:31:08.327 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:08.325+0000 7fe35a238780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:08.328 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:08.327+0000 7fe35a238780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:08.329 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:08.328+0000 7fe35a238780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:08.498 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:08.497+0000 7fe35a238780 -1 Falling back to public interface 2026-02-21T11:31:08.542 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-21T11:31:08.542 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:31:08.542 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:31:08.542 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:31:08.542 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:31:08.543 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:31:08.543 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:08.543 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:31:08.544 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:08.544 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:31:08.766 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:31:08.876 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:08.875+0000 7fe35a238780 -1 osd.0 6 log_to_monitors true 2026-02-21T11:31:09.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:31:09.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:09.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:31:09.768 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:31:09.768 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:09.768 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:31:09.990 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 6 last_clean_interval [5,6) [v2:127.0.0.1:6800/1854969478,v1:127.0.0.1:6801/1854969478] [v2:127.0.0.1:6802/1854969478,v1:127.0.0.1:6803/1854969478] exists,up 840ead89-d67b-4eea-9467-9e25f176bdf6 2026-02-21T11:31:09.991 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:31:09.992 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:31:09.992 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:31:09.992 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-21T11:31:09.992 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T11:31:09.992 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T11:31:09.992 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:09.992 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:09.992 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:09.992 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:09.992 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:09.992 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:09.993 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T11:31:09.993 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T11:31:10.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-21T11:31:10.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-21T11:31:10.039 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2026-02-21T11:31:10.039 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T11:31:10.039 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T11:31:10.040 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:10.040 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:10.040 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:10.040 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:10.040 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:10.040 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:10.040 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T11:31:10.040 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok dump_recovery_reservations 2026-02-21T11:31:10.085 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-21T11:31:10.085 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-21T11:31:10.086 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2026-02-21T11:31:10.086 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T11:31:10.086 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2026-02-21T11:31:10.086 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:10.086 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:10.086 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:10.086 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:10.086 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:10.086 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:10.087 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T11:31:10.087 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok dump_recovery_reservations 2026-02-21T11:31:10.133 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-21T11:31:10.133 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-21T11:31:10.133 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2026-02-21T11:31:10.133 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:10.133 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:10.133 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:10.133 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:10.133 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:10.134 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:10.134 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:10.134 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:10.239 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:10.240 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:10.241 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:10.241 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:10.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:10.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:10.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:10.244 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:10.244 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:10.244 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:10.245 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:10.245 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:10.246 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:10.246 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:10.249 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:10.249 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:10.249 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:10.250 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:10.250 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:10.250 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:10.250 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-21T11:31:10.251 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:10.251 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:10.251 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:10.251 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:10.251 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:10.251 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:10.251 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:10.251 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:10.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:10.253 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:10.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:10.254 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:10.255 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:10.255 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:10.255 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:10.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:10.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:10.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:10.256 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:10.257 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:10.258 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:10.258 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:10.259 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:10.259 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:10.259 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:10.259 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:10.260 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:10.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:10.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-21T11:31:10.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-21T11:31:10.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T11:31:10.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T11:31:10.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:10.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:10.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:10.261 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:10.261 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:10.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:10.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:10.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:10.262 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:10.263 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:10.263 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:10.264 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:10.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:10.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:10.265 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:10.266 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:10.266 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:10.266 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:10.266 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:10.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:10.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:10.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:10.269 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:10.269 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:10.269 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:10.269 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:10.270 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:10.270 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:10.270 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T11:31:10.271 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:31:10.271 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:10.271 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:10.271 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.37640 2026-02-21T11:31:10.272 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:31:10.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:31:10.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:31:10.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T11:31:10.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2026-02-21T11:31:10.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2026-02-21T11:31:10.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2026-02-21T11:31:10.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-21T11:31:10.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:31:10.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:31:10.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:31:10.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-21T11:31:10.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-21T11:31:10.290 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:31:10.290 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:10.290 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:10.290 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:10.291 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:10.291 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:10.291 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:10.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T11:31:10.313 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:31:10.314 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:31:10.314 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:31:10.314 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:31:10.315 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:31:10.316 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:31:10.316 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:31:10.316 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:31:10.316 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:31:10.316 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:10.316 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:10.316 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:10.316 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-mon.a.asok 2026-02-21T11:31:10.316 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:31:10.317 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.37640/ceph-mon.a.asok config get fsid 2026-02-21T11:31:10.358 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:31:10.358 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:31:10.358 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:31:10.359 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:31:10.359 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:31:10.359 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:31:10.359 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:31:10.359 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:31:10.359 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:10.359 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:10.359 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:10.359 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-mon.a.asok 2026-02-21T11:31:10.359 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:31:10.359 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.37640/ceph-mon.a.asok config get mon_host 2026-02-21T11:31:10.398 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2026-02-21T11:31:10.398 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-21T11:31:10.398 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:31:10.398 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:31:10.398 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:31:10.398 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-21T11:31:10.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:31:10.494 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:31:10.494 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:10.494 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:10.495 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:10.495 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:10.495 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:10.495 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:10.495 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:31:10.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:31:10.512 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-21T11:31:10.512 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-21T11:31:10.512 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:31:10.512 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:31:10.512 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:31:10.512 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-21T11:31:10.512 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e6bec3cc-fd12-4810-a327-e5c32c674d23 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-21T11:31:10.512 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:31:10.512 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:31:10.513 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:31:10.513 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-21T11:31:10.513 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-21T11:31:10.513 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:31:10.513 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:31:10.513 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-21T11:31:10.513 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:31:10.513 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:10.513 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:10.513 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:10.513 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:10.513 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:10.513 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:10.514 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:10.514 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:31:10.514 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:31:10.514 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:31:10.514 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-21T11:31:10.514 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-21T11:31:10.514 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:31:10.514 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:31:10.514 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:31:10.514 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:31:10.514 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:31:10.514 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-21T11:31:10.514 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-21T11:31:10.514 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:31:10.515 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=07556e5b-e2a2-4dcd-b7df-868a822637a8 2026-02-21T11:31:10.515 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 07556e5b-e2a2-4dcd-b7df-868a822637a8' 2026-02-21T11:31:10.515 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 07556e5b-e2a2-4dcd-b7df-868a822637a8 2026-02-21T11:31:10.516 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:31:10.521 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB+l5lpwBr8HhAAtajwVd3B7tuGWtv8Mglsfw== 2026-02-21T11:31:10.521 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB+l5lpwBr8HhAAtajwVd3B7tuGWtv8Mglsfw=="}' 2026-02-21T11:31:10.521 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 07556e5b-e2a2-4dcd-b7df-868a822637a8 -i td/mclock-config/0/new.json 2026-02-21T11:31:10.621 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:31:10.627 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-21T11:31:10.627 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e6bec3cc-fd12-4810-a327-e5c32c674d23 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQB+l5lpwBr8HhAAtajwVd3B7tuGWtv8Mglsfw== --osd-uuid 07556e5b-e2a2-4dcd-b7df-868a822637a8 2026-02-21T11:31:10.642 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:10.641+0000 7f4c31e2f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:10.643 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:10.642+0000 7f4c31e2f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:10.644 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:10.643+0000 7f4c31e2f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:10.644 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:10.644+0000 7f4c31e2f780 -1 bdev(0x2d5ac00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:31:10.645 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:10.644+0000 7f4c31e2f780 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-21T11:31:11.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-21T11:31:11.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:31:11.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:31:11.063 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:31:11.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:31:11.164 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:31:11.164 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:31:11.164 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e6bec3cc-fd12-4810-a327-e5c32c674d23 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-21T11:31:11.164 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:31:11.164 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:31:11.164 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:31:11.178 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:11.177+0000 7f154c44d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:11.179 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:11.179+0000 7f154c44d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:11.180 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:11.179+0000 7f154c44d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:11.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:31:11.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:31:11.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:31:11.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:31:11.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:31:11.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:11.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:31:11.262 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:31:11.262 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:11.262 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:31:11.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:31:11.374 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:11.373+0000 7f154c44d780 -1 Falling back to public interface 2026-02-21T11:31:11.677 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:11.675+0000 7f154c44d780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:31:12.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:31:12.358 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:12.358 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:31:12.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:31:12.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:12.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:31:12.452 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2844634873,v1:127.0.0.1:6801/2844634873] [v2:127.0.0.1:6802/2844634873,v1:127.0.0.1:6803/2844634873] exists,up 07556e5b-e2a2-4dcd-b7df-868a822637a8 2026-02-21T11:31:12.452 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:31:12.452 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:31:12.452 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:31:12.452 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2026-02-21T11:31:12.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2026-02-21T11:31:12.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2026-02-21T11:31:12.552 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2026-02-21T11:31:12.552 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-21T11:31:12.552 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:12.552 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:12.552 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:12.552 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:12.553 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:12.553 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:12.553 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-21T11:31:12.553 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_mclock_profile 2026-02-21T11:31:12.598 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2026-02-21T11:31:12.598 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2026-02-21T11:31:12.599 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2026-02-21T11:31:12.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2026-02-21T11:31:12.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2026-02-21T11:31:12.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2026-02-21T11:31:12.658 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:31:12.658 INFO:tasks.workunit.client.0.trial189.stdout: "success": "" 2026-02-21T11:31:12.658 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:31:12.663 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2026-02-21T11:31:12.664 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-21T11:31:12.664 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:12.664 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:12.664 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:12.664 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:12.664 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:12.664 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:12.665 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-21T11:31:12.665 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_mclock_profile 2026-02-21T11:31:12.711 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2026-02-21T11:31:12.711 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2026-02-21T11:31:12.711 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2026-02-21T11:31:12.711 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2026-02-21T11:31:12.711 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2026-02-21T11:31:12.712 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2026-02-21T11:31:12.712 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2026-02-21T11:31:12.712 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-21T11:31:12.712 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:12.712 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:12.714 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:12.714 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:12.714 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:12.714 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:12.714 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-21T11:31:12.714 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T11:31:12.760 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.000000 2026-02-21T11:31:12.761 INFO:tasks.workunit.client.0.trial189.stdout:client_res = 0.000000 2026-02-21T11:31:12.762 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.000000' 2026-02-21T11:31:12.762 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.000000 + 0.1' 2026-02-21T11:31:12.762 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2026-02-21T11:31:12.762 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.100000 2026-02-21T11:31:12.762 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .100000' 2026-02-21T11:31:12.763 INFO:tasks.workunit.client.0.trial189.stdout:client_res_new = .100000 2026-02-21T11:31:12.763 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .100000 2026-02-21T11:31:12.986 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-21T11:31:13.208 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.100000 2026-02-21T11:31:13.208 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.100000 != .100000' 2026-02-21T11:31:13.209 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2026-02-21T11:31:13.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2026-02-21T11:31:13.210 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2026-02-21T11:31:13.211 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2026-02-21T11:31:13.211 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-21T11:31:13.211 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:13.211 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:13.211 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:13.211 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:13.211 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:13.211 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:13.211 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-21T11:31:13.211 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T11:31:13.262 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.100000 2026-02-21T11:31:13.263 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.100000 != .100000' 2026-02-21T11:31:13.263 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2026-02-21T11:31:13.264 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2026-02-21T11:31:13.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2026-02-21T11:31:13.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:13.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:13.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:13.265 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:13.265 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:13.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:13.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:13.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:13.372 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:13.372 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:13.373 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:13.373 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:13.375 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:13.375 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:13.375 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:13.376 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:13.376 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:13.376 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:13.377 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:13.377 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:13.378 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:13.378 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:13.382 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:13.383 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:13.383 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:13.383 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:13.385 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:13.385 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:13.385 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-21T11:31:13.385 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:13.385 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:13.385 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:13.385 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:13.385 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:13.385 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:13.386 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:13.386 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:13.386 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:13.386 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:13.386 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:13.387 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:13.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:13.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:13.388 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:13.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:13.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:13.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:13.389 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:13.390 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:13.390 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:13.391 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:13.392 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:13.392 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:13.392 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:13.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:13.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:13.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:13.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-21T11:31:13.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-21T11:31:13.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T11:31:13.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T11:31:13.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:13.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:13.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:13.394 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:13.395 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:13.395 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:13.395 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:13.395 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:13.395 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:13.395 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:13.396 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:13.396 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:13.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:13.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:13.397 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:13.398 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:13.398 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:13.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:13.399 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:13.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:13.400 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:13.400 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:13.401 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:13.401 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:13.401 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:13.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:13.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:13.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:13.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T11:31:13.403 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:31:13.403 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:13.404 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:13.404 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.37640 2026-02-21T11:31:13.404 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:31:13.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:31:13.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:31:13.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T11:31:13.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2026-02-21T11:31:13.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2026-02-21T11:31:13.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2026-02-21T11:31:13.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T11:31:13.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T11:31:13.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:13.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:13.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:13.407 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:13.407 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:13.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:13.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:13.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:13.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:13.409 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:13.409 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:13.409 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:13.409 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:13.410 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:13.410 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:13.411 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:13.411 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:13.411 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:13.411 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:13.412 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:13.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:13.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:13.414 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:13.414 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:13.414 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:13.414 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:13.415 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:13.415 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:13.415 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T11:31:13.416 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:31:13.416 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:13.416 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:13.416 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.37640 2026-02-21T11:31:13.417 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:31:13.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:31:13.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:31:13.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T11:31:13.419 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2026-02-21T11:31:13.419 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-21T11:31:13.419 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:31:13.419 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:31:13.419 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:31:13.419 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-21T11:31:13.419 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-21T11:31:13.436 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:31:13.436 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:13.436 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:13.436 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:13.437 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:13.437 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:13.437 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:13.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T11:31:13.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:31:13.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:31:13.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:31:13.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:31:13.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:31:13.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:31:13.461 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:31:13.461 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:31:13.461 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:31:13.461 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:13.461 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:13.461 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:13.461 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-mon.a.asok 2026-02-21T11:31:13.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:31:13.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.37640/ceph-mon.a.asok config get fsid 2026-02-21T11:31:13.505 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:31:13.505 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:31:13.505 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:31:13.505 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:31:13.506 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:31:13.506 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:31:13.506 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:31:13.506 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:31:13.506 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:13.506 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:13.506 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:13.506 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-mon.a.asok 2026-02-21T11:31:13.506 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:31:13.506 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.37640/ceph-mon.a.asok config get mon_host 2026-02-21T11:31:13.546 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2026-02-21T11:31:13.546 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-21T11:31:13.546 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:31:13.546 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:31:13.547 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:31:13.547 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-21T11:31:13.547 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:31:13.644 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:31:13.644 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:13.645 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:13.645 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:13.645 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:13.645 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:13.645 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:13.645 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:31:13.645 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:31:13.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-21T11:31:13.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-21T11:31:13.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:31:13.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:31:13.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:31:13.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-21T11:31:13.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e6bec3cc-fd12-4810-a327-e5c32c674d23 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-21T11:31:13.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:31:13.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:31:13.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:31:13.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:31:13.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:31:13.663 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-21T11:31:13.663 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-21T11:31:13.663 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:31:13.663 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=83cbd20a-9119-438d-8abf-1dd8317df268 2026-02-21T11:31:13.663 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 83cbd20a-9119-438d-8abf-1dd8317df268' 2026-02-21T11:31:13.663 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 83cbd20a-9119-438d-8abf-1dd8317df268 2026-02-21T11:31:13.663 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:31:13.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCBl5lpLyjyJxAA3Pu7l6IF3BcfVGbL/mToRw== 2026-02-21T11:31:13.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCBl5lpLyjyJxAA3Pu7l6IF3BcfVGbL/mToRw=="}' 2026-02-21T11:31:13.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 83cbd20a-9119-438d-8abf-1dd8317df268 -i td/mclock-config/0/new.json 2026-02-21T11:31:13.771 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:31:13.777 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-21T11:31:13.778 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e6bec3cc-fd12-4810-a327-e5c32c674d23 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCBl5lpLyjyJxAA3Pu7l6IF3BcfVGbL/mToRw== --osd-uuid 83cbd20a-9119-438d-8abf-1dd8317df268 2026-02-21T11:31:13.793 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:13.792+0000 7fe1fd3d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:13.794 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:13.793+0000 7fe1fd3d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:13.795 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:13.794+0000 7fe1fd3d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:13.796 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:13.794+0000 7fe1fd3d4780 -1 bdev(0x3018c00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:31:13.796 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:13.795+0000 7fe1fd3d4780 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-21T11:31:14.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-21T11:31:14.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:31:14.180 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:31:14.180 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:31:14.180 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:31:14.282 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:31:14.283 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:31:14.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e6bec3cc-fd12-4810-a327-e5c32c674d23 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-21T11:31:14.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:31:14.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:31:14.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:31:14.297 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:14.295+0000 7f52dfe22780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:14.298 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:14.297+0000 7f52dfe22780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:14.299 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:14.298+0000 7f52dfe22780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:14.381 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:31:14.381 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:31:14.381 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:31:14.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:31:14.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:31:14.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:14.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:31:14.382 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:31:14.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:14.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:31:14.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:31:14.501 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:14.500+0000 7f52dfe22780 -1 Falling back to public interface 2026-02-21T11:31:14.907 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:14.906+0000 7f52dfe22780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:31:15.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:31:15.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:15.479 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:31:15.479 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:31:15.479 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:15.479 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:31:15.574 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1095837172,v1:127.0.0.1:6801/1095837172] [v2:127.0.0.1:6802/1095837172,v1:127.0.0.1:6803/1095837172] exists,up 83cbd20a-9119-438d-8abf-1dd8317df268 2026-02-21T11:31:15.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:31:15.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:31:15.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:31:15.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2026-02-21T11:31:15.575 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2026-02-21T11:31:15.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2026-02-21T11:31:15.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2026-02-21T11:31:15.672 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2026-02-21T11:31:15.672 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-21T11:31:15.672 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:15.672 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:15.672 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:15.672 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:15.672 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:15.673 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:15.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-21T11:31:15.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_mclock_profile 2026-02-21T11:31:15.718 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2026-02-21T11:31:15.718 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2026-02-21T11:31:15.718 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2026-02-21T11:31:15.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2026-02-21T11:31:15.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2026-02-21T11:31:15.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2026-02-21T11:31:15.775 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:31:15.775 INFO:tasks.workunit.client.0.trial189.stdout: "success": "" 2026-02-21T11:31:15.775 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:31:15.780 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2026-02-21T11:31:15.780 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-21T11:31:15.780 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:15.780 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:15.780 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:15.780 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:15.780 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:15.780 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:15.781 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-21T11:31:15.781 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_mclock_profile 2026-02-21T11:31:15.829 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2026-02-21T11:31:15.829 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2026-02-21T11:31:15.830 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2026-02-21T11:31:15.830 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2026-02-21T11:31:15.830 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2026-02-21T11:31:15.830 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2026-02-21T11:31:15.831 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-21T11:31:15.831 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2026-02-21T11:31:15.831 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-21T11:31:15.832 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:15.832 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:15.832 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:15.832 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:15.832 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:15.832 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:15.832 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-21T11:31:15.832 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T11:31:15.878 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.000000 2026-02-21T11:31:15.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.000000' 2026-02-21T11:31:15.879 INFO:tasks.workunit.client.0.trial189.stdout:Original client_res for osd.0 = 0.000000 2026-02-21T11:31:15.880 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.000000 + 0.1' 2026-02-21T11:31:15.880 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2026-02-21T11:31:15.880 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.100000 2026-02-21T11:31:15.880 INFO:tasks.workunit.client.0.trial189.stdout:client_res_new = .100000 2026-02-21T11:31:15.881 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .100000' 2026-02-21T11:31:15.881 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .100000 2026-02-21T11:31:16.107 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-21T11:31:16.332 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.100000 2026-02-21T11:31:16.332 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.100000 != .100000' 2026-02-21T11:31:16.332 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2026-02-21T11:31:16.333 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-21T11:31:16.334 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-21T11:31:16.334 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2026-02-21T11:31:16.334 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-21T11:31:16.334 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:16.334 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:16.334 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:16.334 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:16.335 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:16.335 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:16.335 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-21T11:31:16.335 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T11:31:16.380 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.100000 2026-02-21T11:31:16.381 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.100000 != .100000' 2026-02-21T11:31:16.381 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2026-02-21T11:31:16.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-21T11:31:16.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2026-02-21T11:31:16.438 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:31:16.438 INFO:tasks.workunit.client.0.trial189.stdout: "success": "" 2026-02-21T11:31:16.438 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:31:16.442 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2026-02-21T11:31:16.443 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-21T11:31:16.443 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:16.443 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:16.443 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:16.443 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:16.443 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:16.444 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:16.444 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-21T11:31:16.444 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_mclock_profile 2026-02-21T11:31:16.491 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2026-02-21T11:31:16.491 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2026-02-21T11:31:16.491 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2026-02-21T11:31:16.491 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-21T11:31:16.713 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.100000 2026-02-21T11:31:16.714 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.100000 != .100000' 2026-02-21T11:31:16.714 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2026-02-21T11:31:16.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-21T11:31:16.716 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-21T11:31:16.716 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2026-02-21T11:31:16.716 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-21T11:31:16.716 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:16.716 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:16.716 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:16.716 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:16.716 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:16.716 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:16.717 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-21T11:31:16.717 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T11:31:16.763 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.100000 2026-02-21T11:31:16.764 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.100000 != .100000' 2026-02-21T11:31:16.764 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2026-02-21T11:31:16.765 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-21T11:31:16.765 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2026-02-21T11:31:16.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2026-02-21T11:31:21.990 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-21T11:31:22.212 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2026-02-21T11:31:22.213 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2026-02-21T11:31:22.213 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2026-02-21T11:31:22.214 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-21T11:31:22.215 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-21T11:31:22.215 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2026-02-21T11:31:22.215 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-21T11:31:22.215 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:22.215 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:22.215 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:22.215 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:22.215 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:22.216 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:22.216 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-21T11:31:22.216 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-21T11:31:22.262 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.000000 2026-02-21T11:31:22.263 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.000000 != 0.000000' 2026-02-21T11:31:22.263 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2026-02-21T11:31:22.264 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-21T11:31:22.264 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2026-02-21T11:31:22.264 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:22.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:22.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:22.265 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:22.265 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:22.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:22.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:22.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:22.371 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:22.371 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:22.372 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:22.372 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:22.373 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:22.373 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:22.374 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:22.375 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:22.375 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:22.375 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:22.376 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:22.376 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:22.377 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:22.377 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:22.380 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:22.380 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:22.380 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:22.380 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:22.381 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:22.381 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:22.381 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-21T11:31:22.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:22.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:22.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:22.382 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:22.382 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:22.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:22.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:22.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:22.384 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:22.384 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:22.385 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:22.385 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:22.386 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:22.386 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:22.386 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:22.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:22.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:22.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:22.387 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:22.388 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:22.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:22.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:22.390 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:22.390 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:22.390 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:22.390 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:22.391 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:22.391 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:22.391 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-21T11:31:22.391 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-21T11:31:22.391 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T11:31:22.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T11:31:22.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:22.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:22.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:22.392 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:22.392 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:22.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:22.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:22.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:22.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:22.394 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:22.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:22.395 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:22.396 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:22.396 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:22.396 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:22.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:22.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:22.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:22.397 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:22.398 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:22.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:22.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:22.400 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:22.400 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:22.400 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:22.400 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:22.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:22.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:22.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T11:31:22.402 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:31:22.402 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:22.402 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:22.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.37640 2026-02-21T11:31:22.403 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:31:22.405 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:31:22.405 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:31:22.405 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T11:31:22.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2026-02-21T11:31:22.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2026-02-21T11:31:22.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2026-02-21T11:31:22.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-21T11:31:22.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-21T11:31:22.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:22.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:22.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:22.406 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:22.406 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:22.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:22.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:22.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:22.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:22.407 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:22.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:22.408 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:22.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:22.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:22.409 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:22.409 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:22.409 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:22.410 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:22.410 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:22.410 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:22.411 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:22.411 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:22.413 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:22.413 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:22.413 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:22.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:22.414 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:22.414 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:22.414 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-21T11:31:22.415 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:31:22.415 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:22.415 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:22.415 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.37640 2026-02-21T11:31:22.416 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:31:22.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:31:22.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:31:22.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-21T11:31:22.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2026-02-21T11:31:22.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-21T11:31:22.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:31:22.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:31:22.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:31:22.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-21T11:31:22.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-21T11:31:22.435 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:31:22.435 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:22.435 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:22.435 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:22.435 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:22.435 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:22.435 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:22.435 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T11:31:22.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:31:22.459 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:31:22.459 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:31:22.459 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:31:22.459 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:31:22.459 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:31:22.460 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:31:22.460 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:31:22.460 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:31:22.461 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:22.461 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:22.461 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:22.461 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-mon.a.asok 2026-02-21T11:31:22.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:31:22.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.37640/ceph-mon.a.asok config get fsid 2026-02-21T11:31:22.504 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:31:22.504 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:31:22.504 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:31:22.504 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:31:22.504 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:31:22.504 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:31:22.504 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:31:22.504 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:31:22.504 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:22.504 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:22.505 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:22.505 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-mon.a.asok 2026-02-21T11:31:22.505 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:31:22.505 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.37640/ceph-mon.a.asok config get mon_host 2026-02-21T11:31:22.544 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2026-02-21T11:31:22.544 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-21T11:31:22.545 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:31:22.545 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:31:22.545 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:31:22.545 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-21T11:31:22.545 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:31:22.641 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:31:22.642 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:22.642 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:22.642 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:22.642 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:22.642 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:22.642 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:22.643 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:31:22.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e6bec3cc-fd12-4810-a327-e5c32c674d23 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:22.658 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ce4887f4-0d20-413f-a134-88c19e874221 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ce4887f4-0d20-413f-a134-88c19e874221' 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 ce4887f4-0d20-413f-a134-88c19e874221 2026-02-21T11:31:22.659 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:31:22.668 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCKl5lpalTAJxAAVYPMzqJFG0caPjejx6J7bA== 2026-02-21T11:31:22.668 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCKl5lpalTAJxAAVYPMzqJFG0caPjejx6J7bA=="}' 2026-02-21T11:31:22.668 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ce4887f4-0d20-413f-a134-88c19e874221 -i td/mclock-config/0/new.json 2026-02-21T11:31:22.768 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:31:22.774 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-21T11:31:22.775 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e6bec3cc-fd12-4810-a327-e5c32c674d23 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCKl5lpalTAJxAAVYPMzqJFG0caPjejx6J7bA== --osd-uuid ce4887f4-0d20-413f-a134-88c19e874221 2026-02-21T11:31:22.791 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:22.790+0000 7f5971235780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:22.792 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:22.791+0000 7f5971235780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:22.793 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:22.792+0000 7f5971235780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:22.794 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:22.792+0000 7f5971235780 -1 bdev(0x258ec00 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:31:22.794 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:22.792+0000 7f5971235780 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-21T11:31:23.193 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-21T11:31:23.194 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:31:23.194 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:31:23.195 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:31:23.195 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:31:23.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:31:23.296 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:31:23.297 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e6bec3cc-fd12-4810-a327-e5c32c674d23 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.37640/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-21T11:31:23.297 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:31:23.297 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:31:23.297 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:31:23.309 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:23.307+0000 7f011a042780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:23.310 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:23.309+0000 7f011a042780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:23.311 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:23.310+0000 7f011a042780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:23.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:31:23.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:31:23.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:31:23.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:31:23.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:31:23.393 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:23.393 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:31:23.393 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:31:23.393 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:23.393 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:31:23.488 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:31:23.518 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:23.516+0000 7f011a042780 -1 Falling back to public interface 2026-02-21T11:31:23.894 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:23.892+0000 7f011a042780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:31:24.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:31:24.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:24.490 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:31:24.490 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:31:24.490 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:24.490 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:31:24.586 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1504803424,v1:127.0.0.1:6801/1504803424] [v2:127.0.0.1:6802/1504803424,v1:127.0.0.1:6803/1504803424] exists,up ce4887f4-0d20-413f-a134-88c19e874221 2026-02-21T11:31:24.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:31:24.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:31:24.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:31:24.587 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T11:31:24.587 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:24.587 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:24.587 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:24.588 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:24.588 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:24.588 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:24.588 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T11:31:24.588 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_recovery_max_active 2026-02-21T11:31:24.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2026-02-21T11:31:24.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2026-02-21T11:31:24.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2026-02-21T11:31:24.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2026-02-21T11:31:24.740 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2026-02-21T11:31:26.743 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T11:31:26.743 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:26.743 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:26.743 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:26.743 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:26.743 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:26.743 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:26.744 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T11:31:26.744 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_recovery_max_active 2026-02-21T11:31:26.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2026-02-21T11:31:26.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2026-02-21T11:31:26.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2026-02-21T11:31:27.016 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2026-02-21T11:31:27.239 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2026-02-21T11:31:29.242 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2026-02-21T11:31:29.242 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:31:29.242 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:31:29.243 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:29.243 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:29.243 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:29.243 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.37640/ceph-osd.0.asok 2026-02-21T11:31:29.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2026-02-21T11:31:29.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.37640/ceph-osd.0.asok config get osd_recovery_max_active 2026-02-21T11:31:29.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2026-02-21T11:31:29.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2026-02-21T11:31:29.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2026-02-21T11:31:29.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:29.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:29.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:29.291 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:29.291 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:29.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:29.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:29.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:29.398 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:29.399 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:29.400 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:29.400 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:29.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:29.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:29.402 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:29.404 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:29.404 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:29.404 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:29.404 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:29.405 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:29.405 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:29.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:29.409 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:29.409 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:29.409 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:29.409 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:29.412 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:29.412 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:29.412 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-21T11:31:29.412 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:29.412 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:29.412 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:29.413 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:29.413 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:29.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:29.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:29.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:29.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:29.414 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:29.415 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:29.415 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:29.415 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:29.415 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:29.415 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:29.416 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:29.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:29.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:29.417 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:29.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:29.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:29.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:29.421 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:29.421 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:29.421 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:29.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:29.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:29.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:29.422 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: main: code=0 2026-02-21T11:31:29.422 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2433: main: teardown td/mclock-config 0 2026-02-21T11:31:29.422 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-21T11:31:29.422 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-21T11:31:29.422 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-21T11:31:29.422 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:29.422 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:29.422 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:29.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:29.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:29.424 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:29.424 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:29.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:29.425 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:29.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:29.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:29.425 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:29.426 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:29.426 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:29.426 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:29.426 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:29.426 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:29.427 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-21T11:31:29.427 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-21T11:31:29.428 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:29.428 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:29.428 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.37640 2026-02-21T11:31:29.428 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.37640 2026-02-21T11:31:29.430 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:29.430 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:29.430 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: main: return 0 2026-02-21T11:31:29.431 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-21T11:31:29.431 DEBUG:teuthology.orchestra.run.trial189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-21T11:31:29.495 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2026-02-21T11:31:29.495 DEBUG:teuthology.orchestra.run.trial189:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2026-02-21T11:31:29.557 INFO:tasks.workunit.client.0.trial189.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: main: export PYTHONWARNINGS=ignore 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: main: PYTHONWARNINGS=ignore 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: main: export CEPH_CONF=/dev/null 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: main: CEPH_CONF=/dev/null 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: main: unset CEPH_ARGS 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: main: local code 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: main: run td/network-ping 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2026-02-21T11:31:29.560 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2026-02-21T11:31:29.561 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none ' 2026-02-21T11:31:29.561 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2026-02-21T11:31:29.561 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2026-02-21T11:31:29.561 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2026-02-21T11:31:29.561 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2026-02-21T11:31:29.561 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2026-02-21T11:31:29.562 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-21T11:31:29.563 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2026-02-21T11:31:29.563 INFO:tasks.workunit.client.0.trial189.stderr:TEST_network_ping_test2' 2026-02-21T11:31:29.563 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2026-02-21T11:31:29.563 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2026-02-21T11:31:29.563 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/network-ping 2026-02-21T11:31:29.563 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/network-ping 2026-02-21T11:31:29.563 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-21T11:31:29.563 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:31:29.563 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-21T11:31:29.563 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:31:29.563 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:31:29.564 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:31:29.564 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:31:29.564 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:31:29.565 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:31:29.565 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:31:29.566 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:31:29.566 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:31:29.567 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:31:29.567 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:31:29.567 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:31:29.569 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:29.569 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:31:29.569 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:31:29.569 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:31:29.569 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:31:29.570 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:31:29.570 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-21T11:31:29.571 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:31:29.571 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:29.572 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:31:29.572 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.47438 2026-02-21T11:31:29.572 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:31:29.572 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:31:29.573 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/network-ping 2026-02-21T11:31:29.573 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:31:29.574 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:29.574 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:31:29.574 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.47438 2026-02-21T11:31:29.576 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:31:29.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-21T11:31:29.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-21T11:31:29.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:31:29.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2026-02-21T11:31:29.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2026-02-21T11:31:29.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2026-02-21T11:31:29.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2026-02-21T11:31:29.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/network-ping 2026-02-21T11:31:29.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:31:29.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:31:29.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:31:29.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/network-ping/a 2026-02-21T11:31:29.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2026-02-21T11:31:29.593 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:31:29.594 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:29.594 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:29.594 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:29.594 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:29.594 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:31:29.594 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:31:29.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T11:31:29.616 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:31:29.618 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:31:29.618 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:31:29.618 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:31:29.618 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:31:29.618 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:31:29.618 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:31:29.618 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:31:29.618 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:31:29.618 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:29.618 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:29.618 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:31:29.619 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-mon.a.asok 2026-02-21T11:31:29.619 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:31:29.619 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.47438/ceph-mon.a.asok config get fsid 2026-02-21T11:31:29.662 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:31:29.662 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:31:29.662 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:31:29.662 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:31:29.662 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:31:29.662 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:31:29.662 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:31:29.662 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:31:29.662 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:31:29.663 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:29.663 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:31:29.663 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-mon.a.asok 2026-02-21T11:31:29.663 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:31:29.663 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.47438/ceph-mon.a.asok config get mon_host 2026-02-21T11:31:29.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2026-02-21T11:31:29.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/network-ping 2026-02-21T11:31:29.703 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:31:29.703 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:31:29.703 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:31:29.703 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/network-ping/x 2026-02-21T11:31:29.703 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:31:29.800 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:31:29.801 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:29.801 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:29.801 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:29.801 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:29.801 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:31:29.801 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:31:29.801 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:31:29.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/0 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:29.818 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:31:29.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/0 2026-02-21T11:31:29.820 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:31:29.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dcfb1953-efc8-4573-98b2-9d43475fec8d 2026-02-21T11:31:29.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 dcfb1953-efc8-4573-98b2-9d43475fec8d' 2026-02-21T11:31:29.821 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 dcfb1953-efc8-4573-98b2-9d43475fec8d 2026-02-21T11:31:29.821 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:31:29.831 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCRl5lpYtxwMRAA7sHGNYm2dF/uo0uEtI0/5g== 2026-02-21T11:31:29.831 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCRl5lpYtxwMRAA7sHGNYm2dF/uo0uEtI0/5g=="}' 2026-02-21T11:31:29.831 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dcfb1953-efc8-4573-98b2-9d43475fec8d -i td/network-ping/0/new.json 2026-02-21T11:31:29.931 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:31:29.937 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/0/new.json 2026-02-21T11:31:29.938 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCRl5lpYtxwMRAA7sHGNYm2dF/uo0uEtI0/5g== --osd-uuid dcfb1953-efc8-4573-98b2-9d43475fec8d 2026-02-21T11:31:29.954 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:29.952+0000 7f2b12dd6780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:29.955 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:29.953+0000 7f2b12dd6780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:29.956 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:29.954+0000 7f2b12dd6780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:29.956 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:29.954+0000 7f2b12dd6780 -1 bdev(0x2d4cc00 td/network-ping/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:31:29.956 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:29.954+0000 7f2b12dd6780 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2026-02-21T11:31:30.388 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/0/keyring 2026-02-21T11:31:30.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:31:30.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:31:30.390 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:31:30.390 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:31:30.494 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:31:30.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:31:30.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:31:30.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:31:30.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:31:30.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:31:30.509 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:30.507+0000 7f704904a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:30.511 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:30.508+0000 7f704904a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:30.512 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:30.509+0000 7f704904a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:30.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:31:30.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:31:30.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:31:30.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:31:30.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:31:30.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:30.593 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:31:30.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:31:30.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:30.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:31:30.688 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:31:30.745 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:30.742+0000 7f704904a780 -1 Falling back to public interface 2026-02-21T11:31:31.049 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:31.046+0000 7f704904a780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:31:31.689 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:31:31.689 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:31.690 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:31:31.691 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:31:31.691 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:31.691 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:31:31.789 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2907466460,v1:127.0.0.1:6801/2907466460] [v2:127.0.0.1:6802/2907466460,v1:127.0.0.1:6803/2907466460] exists,up dcfb1953-efc8-4573-98b2-9d43475fec8d 2026-02-21T11:31:31.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:31:31.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:31:31.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:31:31.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2026-02-21T11:31:31.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-21T11:31:31.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:31:31.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T11:31:31.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:31:31.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/1 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:31:31.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:31:31.792 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:31:31.792 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-21T11:31:31.792 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-21T11:31:31.792 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:31:31.792 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:31:31.792 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:31:31.792 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:31:31.792 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:31:31.792 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:31:31.792 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/1 2026-02-21T11:31:31.792 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:31:31.792 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=41263ce1-36cb-43c6-a732-4b4d8e521c24 2026-02-21T11:31:31.792 INFO:tasks.workunit.client.0.trial189.stdout:add osd1 41263ce1-36cb-43c6-a732-4b4d8e521c24 2026-02-21T11:31:31.793 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 41263ce1-36cb-43c6-a732-4b4d8e521c24' 2026-02-21T11:31:31.793 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:31:31.803 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCTl5lpt87GLxAAvpk4zGAqiMFVpf2MvW219A== 2026-02-21T11:31:31.803 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCTl5lpt87GLxAAvpk4zGAqiMFVpf2MvW219A=="}' 2026-02-21T11:31:31.803 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 41263ce1-36cb-43c6-a732-4b4d8e521c24 -i td/network-ping/1/new.json 2026-02-21T11:31:31.906 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:31:31.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/1/new.json 2026-02-21T11:31:31.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTl5lpt87GLxAAvpk4zGAqiMFVpf2MvW219A== --osd-uuid 41263ce1-36cb-43c6-a732-4b4d8e521c24 2026-02-21T11:31:31.928 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:31.926+0000 7f2b9a422780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:31.930 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:31.928+0000 7f2b9a422780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:31.931 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:31.929+0000 7f2b9a422780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:31.931 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:31.929+0000 7f2b9a422780 -1 bdev(0x3df4c00 td/network-ping/1/block) open stat got: (1) Operation not permitted 2026-02-21T11:31:31.932 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:31.929+0000 7f2b9a422780 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2026-02-21T11:31:32.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/1/keyring 2026-02-21T11:31:32.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:31:32.257 INFO:tasks.workunit.client.0.trial189.stdout:adding osd1 key to auth repository 2026-02-21T11:31:32.257 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T11:31:32.257 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:31:32.572 INFO:tasks.workunit.client.0.trial189.stdout:start osd.1 2026-02-21T11:31:32.572 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T11:31:32.572 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:31:32.572 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:31:32.573 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:31:32.573 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:31:32.587 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:32.584+0000 7f429923a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:32.588 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:32.586+0000 7f429923a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:32.589 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:32.587+0000 7f429923a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:32.782 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:32.780+0000 7f429923a780 -1 Falling back to public interface 2026-02-21T11:31:32.796 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T11:31:32.796 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:31:32.796 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T11:31:32.796 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:31:32.796 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:31:32.796 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:32.797 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:31:32.797 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:31:32.797 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:32.798 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:31:33.018 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:31:33.110 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:33.107+0000 7f429923a780 -1 osd.1 0 log_to_monitors true 2026-02-21T11:31:34.019 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:31:34.019 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:34.020 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:31:34.020 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:31:34.021 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:34.021 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:31:34.245 INFO:tasks.workunit.client.0.trial189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3002309683,v1:127.0.0.1:6811/3002309683] [v2:127.0.0.1:6812/3002309683,v1:127.0.0.1:6813/3002309683] exists,up 41263ce1-36cb-43c6-a732-4b4d8e521c24 2026-02-21T11:31:34.247 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:31:34.247 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:31:34.247 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:31:34.247 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2026-02-21T11:31:34.247 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-21T11:31:34.247 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/2 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/2 2026-02-21T11:31:34.248 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:31:34.249 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ca081319-f820-46a9-9a40-e64e7b423a77 2026-02-21T11:31:34.249 INFO:tasks.workunit.client.0.trial189.stdout:add osd2 ca081319-f820-46a9-9a40-e64e7b423a77 2026-02-21T11:31:34.249 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 ca081319-f820-46a9-9a40-e64e7b423a77' 2026-02-21T11:31:34.249 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:31:34.258 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCWl5lplwtUDxAArsa13Cfq+AHsOTaMtijqAg== 2026-02-21T11:31:34.259 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCWl5lplwtUDxAArsa13Cfq+AHsOTaMtijqAg=="}' 2026-02-21T11:31:34.259 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ca081319-f820-46a9-9a40-e64e7b423a77 -i td/network-ping/2/new.json 2026-02-21T11:31:34.485 INFO:tasks.workunit.client.0.trial189.stdout:2 2026-02-21T11:31:34.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/2/new.json 2026-02-21T11:31:34.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCWl5lplwtUDxAArsa13Cfq+AHsOTaMtijqAg== --osd-uuid ca081319-f820-46a9-9a40-e64e7b423a77 2026-02-21T11:31:34.508 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:34.506+0000 7f7144a0f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:34.509 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:34.507+0000 7f7144a0f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:34.510 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:34.508+0000 7f7144a0f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:34.511 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:34.508+0000 7f7144a0f780 -1 bdev(0x247cc00 td/network-ping/2/block) open stat got: (1) Operation not permitted 2026-02-21T11:31:34.511 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:34.509+0000 7f7144a0f780 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2026-02-21T11:31:34.836 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/2/keyring 2026-02-21T11:31:34.837 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:31:34.837 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T11:31:34.838 INFO:tasks.workunit.client.0.trial189.stdout:adding osd2 key to auth repository 2026-02-21T11:31:34.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:31:35.156 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T11:31:35.156 INFO:tasks.workunit.client.0.trial189.stdout:start osd.2 2026-02-21T11:31:35.158 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:31:35.158 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:31:35.158 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:31:35.158 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:31:35.170 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:35.167+0000 7f52e9840780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:35.171 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:35.169+0000 7f52e9840780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:35.172 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:35.169+0000 7f52e9840780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:31:35.311 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:35.308+0000 7f52e9840780 -1 Falling back to public interface 2026-02-21T11:31:35.383 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T11:31:35.383 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:31:35.383 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T11:31:35.383 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:31:35.383 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:31:35.383 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:35.383 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:31:35.383 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:31:35.384 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:35.384 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T11:31:35.608 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:31:35.606+0000 7f52e9840780 -1 osd.2 0 log_to_monitors true 2026-02-21T11:31:35.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:31:36.613 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:31:36.613 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:31:36.613 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:31:36.614 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:31:36.614 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:31:36.614 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T11:31:36.838 INFO:tasks.workunit.client.0.trial189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1313768420,v1:127.0.0.1:6819/1313768420] [v2:127.0.0.1:6820/1313768420,v1:127.0.0.1:6821/1313768420] exists,up ca081319-f820-46a9-9a40-e64e7b423a77 2026-02-21T11:31:36.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:31:36.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:31:36.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:31:36.839 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2026-02-21T11:31:41.839 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2026-02-21T11:31:41.840 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2026-02-21T11:31:42.117 INFO:tasks.workunit.client.0.trial189.stderr:pool 'foo' already exists 2026-02-21T11:31:42.125 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:31:43.126 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2026-02-21T11:31:43.146 INFO:tasks.workunit.client.0.trial189.stdout:hints = 1 2026-02-21T11:31:43.146 INFO:tasks.workunit.client.0.trial189.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2026-02-21T11:31:43.146 INFO:tasks.workunit.client.0.trial189.stdout:Object prefix: benchmark_data_trial189_51941 2026-02-21T11:31:43.146 INFO:tasks.workunit.client.0.trial189.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-21T11:31:43.146 INFO:tasks.workunit.client.0.trial189.stdout: 0 0 0 0 0 0 - 0 2026-02-21T11:31:44.147 INFO:tasks.workunit.client.0.trial189.stdout: 1 15 11341 11326 44.2387 44.2422 0.000952697 0.00141037 2026-02-21T11:31:45.147 INFO:tasks.workunit.client.0.trial189.stdout: 2 16 24138 24122 47.1086 49.9844 0.00102366 0.00132506 2026-02-21T11:31:46.147 INFO:tasks.workunit.client.0.trial189.stdout: 3 16 35787 35771 46.5725 45.5039 0.00100007 0.0013406 2026-02-21T11:31:47.147 INFO:tasks.workunit.client.0.trial189.stdout: 4 16 47869 47853 46.7273 47.1953 0.00111148 0.00133617 2026-02-21T11:31:48.147 INFO:tasks.workunit.client.0.trial189.stdout: 5 16 60243 60227 47.0486 48.3359 0.000926387 0.00132704 2026-02-21T11:31:49.147 INFO:tasks.workunit.client.0.trial189.stdout: 6 16 72243 72227 47.0191 46.875 0.00230389 0.00132789 2026-02-21T11:31:50.147 INFO:tasks.workunit.client.0.trial189.stdout: 7 16 83505 83489 46.5862 43.9922 0.0021708 0.00134028 2026-02-21T11:31:51.147 INFO:tasks.workunit.client.0.trial189.stdout: 8 16 94132 94116 45.9515 41.5117 0.00155868 0.00135879 2026-02-21T11:31:52.147 INFO:tasks.workunit.client.0.trial189.stdout: 9 16 104768 104752 45.4617 41.5469 0.0018394 0.00137349 2026-02-21T11:31:53.147 INFO:tasks.workunit.client.0.trial189.stdout: 10 4 115262 115258 45.0191 41.0391 0.00131349 0.00138702 2026-02-21T11:31:54.147 INFO:tasks.workunit.client.0.trial189.stdout:Total time run: 10.001 2026-02-21T11:31:54.147 INFO:tasks.workunit.client.0.trial189.stdout:Total writes made: 115262 2026-02-21T11:31:54.147 INFO:tasks.workunit.client.0.trial189.stdout:Write size: 4096 2026-02-21T11:31:54.148 INFO:tasks.workunit.client.0.trial189.stdout:Object size: 4096 2026-02-21T11:31:54.148 INFO:tasks.workunit.client.0.trial189.stdout:Bandwidth (MB/sec): 45.0199 2026-02-21T11:31:54.148 INFO:tasks.workunit.client.0.trial189.stdout:Stddev Bandwidth: 3.0899 2026-02-21T11:31:54.148 INFO:tasks.workunit.client.0.trial189.stdout:Max bandwidth (MB/sec): 49.9844 2026-02-21T11:31:54.148 INFO:tasks.workunit.client.0.trial189.stdout:Min bandwidth (MB/sec): 41.0391 2026-02-21T11:31:54.148 INFO:tasks.workunit.client.0.trial189.stdout:Average IOPS: 11525 2026-02-21T11:31:54.148 INFO:tasks.workunit.client.0.trial189.stdout:Stddev IOPS: 791.015 2026-02-21T11:31:54.148 INFO:tasks.workunit.client.0.trial189.stdout:Max IOPS: 12796 2026-02-21T11:31:54.148 INFO:tasks.workunit.client.0.trial189.stdout:Min IOPS: 10506 2026-02-21T11:31:54.148 INFO:tasks.workunit.client.0.trial189.stdout:Average Latency(s): 0.00138701 2026-02-21T11:31:54.148 INFO:tasks.workunit.client.0.trial189.stdout:Stddev Latency(s): 0.000691105 2026-02-21T11:31:54.148 INFO:tasks.workunit.client.0.trial189.stdout:Max latency(s): 0.0119041 2026-02-21T11:31:54.148 INFO:tasks.workunit.client.0.trial189.stdout:Min latency(s): 0.000447935 2026-02-21T11:31:54.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2026-02-21T11:32:04.153 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2026-02-21T11:32:04.153 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:32:04.154 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:32:04.429 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:32:04.429 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:32:04.429 INFO:tasks.workunit.client.0.trial189.stderr:2' 2026-02-21T11:32:04.429 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:32:04.429 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:32:04.430 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:32:04.595 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836489 2026-02-21T11:32:04.595 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836489 2026-02-21T11:32:04.595 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836489' 2026-02-21T11:32:04.595 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:32:04.596 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:32:04.713 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=42949672969 2026-02-21T11:32:04.713 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 42949672969 2026-02-21T11:32:04.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836489 1-42949672969' 2026-02-21T11:32:04.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:32:04.714 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-21T11:32:04.813 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=60129542152 2026-02-21T11:32:04.813 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 60129542152 2026-02-21T11:32:04.813 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836489 1-42949672969 2-60129542152' 2026-02-21T11:32:04.813 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:32:04.814 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836489 2026-02-21T11:32:04.814 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:32:04.814 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:32:04.815 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836489 2026-02-21T11:32:04.815 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:32:04.816 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836489 2026-02-21T11:32:04.816 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836489 2026-02-21T11:32:04.816 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-21T11:32:04.817 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:32:05.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836487 -lt 21474836489 2026-02-21T11:32:05.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:32:06.099 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:32:06.099 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:32:06.434 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-21T11:32:06.434 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:32:06.434 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-42949672969 2026-02-21T11:32:06.434 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:32:06.435 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:32:06.436 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-42949672969 2026-02-21T11:32:06.436 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:32:06.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=42949672969 2026-02-21T11:32:06.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2026-02-21T11:32:06.437 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 42949672969 2026-02-21T11:32:06.437 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:32:06.697 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 42949672969 -lt 42949672969 2026-02-21T11:32:06.697 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:32:06.697 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 2-60129542152 2026-02-21T11:32:06.698 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:32:06.698 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=2 2026-02-21T11:32:06.699 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 2-60129542152 2026-02-21T11:32:06.699 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:32:06.700 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=60129542152 2026-02-21T11:32:06.700 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.2 seq 60129542152' 2026-02-21T11:32:06.701 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.2 seq 60129542152 2026-02-21T11:32:06.701 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T11:32:06.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 60129542152 -lt 60129542152 2026-02-21T11:32:06.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T11:32:06.928 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2026-02-21T11:32:06.928 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:32:06.928 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:32:06.929 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:32:06.929 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:32:06.929 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:32:06.929 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-osd.0.asok 2026-02-21T11:32:06.929 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T11:32:06.929 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.47438/ceph-osd.0.asok dump_osd_network 2026-02-21T11:32:06.971 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:32:06.971 INFO:tasks.workunit.client.0.trial189.stdout: "threshold": 1000, 2026-02-21T11:32:06.971 INFO:tasks.workunit.client.0.trial189.stdout: "entries": [] 2026-02-21T11:32:06.971 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:32:06.975 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:06.975 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T11:32:06.977 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2026-02-21T11:32:06.977 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:06.977 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2026-02-21T11:32:06.979 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2026-02-21T11:32:06.980 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T11:32:06.980 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-21T11:32:06.980 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-21T11:32:06.980 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-21T11:32:06.980 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:32:06.981 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:32:06.981 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:32:06.981 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-mgr.x.asok 2026-02-21T11:32:06.981 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T11:32:06.981 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.47438/ceph-mgr.x.asok dump_osd_network 2026-02-21T11:32:07.020 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:32:07.020 INFO:tasks.workunit.client.0.trial189.stdout: "threshold": 1000, 2026-02-21T11:32:07.020 INFO:tasks.workunit.client.0.trial189.stdout: "entries": [] 2026-02-21T11:32:07.020 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:32:07.024 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:07.025 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T11:32:07.026 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2026-02-21T11:32:07.027 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:07.027 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2026-02-21T11:32:07.029 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2026-02-21T11:32:07.029 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T11:32:07.031 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2026-02-21T11:32:07.031 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:32:07.031 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:32:07.031 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:32:07.031 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:32:07.031 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:32:07.031 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-osd.0.asok 2026-02-21T11:32:07.032 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T11:32:07.032 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.47438/ceph-osd.0.asok dump_osd_network 0 2026-02-21T11:32:07.073 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:32:07.073 INFO:tasks.workunit.client.0.trial189.stdout: "threshold": 0, 2026-02-21T11:32:07.073 INFO:tasks.workunit.client.0.trial189.stdout: "entries": [ 2026-02-21T11:32:07.073 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.073 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:05 2026", 2026-02-21T11:32:07.074 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.074 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:07.074 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:32:07.074 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:07.074 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.074 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.130, 2026-02-21T11:32:07.074 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.159, 2026-02-21T11:32:07.074 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.304 2026-02-21T11:32:07.074 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.074 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.074 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.214, 2026-02-21T11:32:07.074 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.371, 2026-02-21T11:32:07.075 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 1.305 2026-02-21T11:32:07.075 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.075 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.075 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.214, 2026-02-21T11:32:07.075 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.371, 2026-02-21T11:32:07.075 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 1.305 2026-02-21T11:32:07.075 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.075 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.205 2026-02-21T11:32:07.075 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.075 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.075 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:05 2026", 2026-02-21T11:32:07.076 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.076 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:07.076 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:32:07.076 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:07.076 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.076 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.132, 2026-02-21T11:32:07.076 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.155, 2026-02-21T11:32:07.076 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.190 2026-02-21T11:32:07.076 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.076 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.076 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.223, 2026-02-21T11:32:07.077 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.784, 2026-02-21T11:32:07.077 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.784 2026-02-21T11:32:07.077 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.077 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.077 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.223, 2026-02-21T11:32:07.077 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.784, 2026-02-21T11:32:07.077 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.784 2026-02-21T11:32:07.077 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.077 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.166 2026-02-21T11:32:07.077 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.077 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.077 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:03 2026", 2026-02-21T11:32:07.077 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.078 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:07.078 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:32:07.078 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:07.078 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.078 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.119, 2026-02-21T11:32:07.078 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.151, 2026-02-21T11:32:07.078 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.173 2026-02-21T11:32:07.078 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.078 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.078 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.162, 2026-02-21T11:32:07.078 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.304, 2026-02-21T11:32:07.078 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.304 2026-02-21T11:32:07.078 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.079 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.079 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.162, 2026-02-21T11:32:07.079 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.304, 2026-02-21T11:32:07.079 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.304 2026-02-21T11:32:07.079 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.079 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.192 2026-02-21T11:32:07.079 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.079 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.079 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:03 2026", 2026-02-21T11:32:07.079 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.079 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:07.079 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:32:07.079 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:07.080 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.080 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.111, 2026-02-21T11:32:07.080 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.148, 2026-02-21T11:32:07.080 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.162 2026-02-21T11:32:07.080 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.080 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.080 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.142, 2026-02-21T11:32:07.080 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.269, 2026-02-21T11:32:07.080 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.269 2026-02-21T11:32:07.080 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.080 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.080 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.142, 2026-02-21T11:32:07.080 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.269, 2026-02-21T11:32:07.081 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.269 2026-02-21T11:32:07.081 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.081 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.182 2026-02-21T11:32:07.081 INFO:tasks.workunit.client.0.trial189.stdout: } 2026-02-21T11:32:07.081 INFO:tasks.workunit.client.0.trial189.stdout: ] 2026-02-21T11:32:07.081 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:32:07.082 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:07.082 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T11:32:07.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2026-02-21T11:32:07.082 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:07.082 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2026-02-21T11:32:07.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2026-02-21T11:32:07.084 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T11:32:07.084 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-21T11:32:07.084 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-21T11:32:07.084 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-21T11:32:07.084 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:32:07.084 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:32:07.084 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:32:07.085 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-mgr.x.asok 2026-02-21T11:32:07.085 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T11:32:07.085 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.47438/ceph-mgr.x.asok dump_osd_network 0 2026-02-21T11:32:07.120 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:32:07.120 INFO:tasks.workunit.client.0.trial189.stdout: "threshold": 0, 2026-02-21T11:32:07.120 INFO:tasks.workunit.client.0.trial189.stdout: "entries": [ 2026-02-21T11:32:07.121 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.121 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:00 2026", 2026-02-21T11:32:07.121 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.121 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:07.121 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:32:07.121 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:07.121 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.121 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.127, 2026-02-21T11:32:07.121 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.208, 2026-02-21T11:32:07.121 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.320 2026-02-21T11:32:07.121 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.122 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.122 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.111, 2026-02-21T11:32:07.122 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.081, 2026-02-21T11:32:07.122 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.081 2026-02-21T11:32:07.122 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.122 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.122 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.150, 2026-02-21T11:32:07.122 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 1.305, 2026-02-21T11:32:07.122 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 1.305 2026-02-21T11:32:07.122 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.122 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.116 2026-02-21T11:32:07.123 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.123 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.123 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:01 2026", 2026-02-21T11:32:07.123 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.123 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 2, 2026-02-21T11:32:07.123 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:32:07.123 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:07.123 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.123 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.106, 2026-02-21T11:32:07.123 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.233, 2026-02-21T11:32:07.123 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.182 2026-02-21T11:32:07.123 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.124 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.124 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.096, 2026-02-21T11:32:07.124 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.096, 2026-02-21T11:32:07.124 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.096 2026-02-21T11:32:07.124 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.124 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.124 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.127, 2026-02-21T11:32:07.124 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 1.520, 2026-02-21T11:32:07.124 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 1.520 2026-02-21T11:32:07.124 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.124 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.169 2026-02-21T11:32:07.124 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.124 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.125 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:01 2026", 2026-02-21T11:32:07.125 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.125 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 2, 2026-02-21T11:32:07.125 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:32:07.125 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:07.125 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.125 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.115, 2026-02-21T11:32:07.125 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.222, 2026-02-21T11:32:07.125 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.168 2026-02-21T11:32:07.125 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.125 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.125 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.090, 2026-02-21T11:32:07.125 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.090, 2026-02-21T11:32:07.126 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.090 2026-02-21T11:32:07.126 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.126 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.126 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.145, 2026-02-21T11:32:07.126 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 1.537, 2026-02-21T11:32:07.126 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 1.537 2026-02-21T11:32:07.126 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.126 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.154 2026-02-21T11:32:07.126 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.126 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.126 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:00 2026", 2026-02-21T11:32:07.126 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.127 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:07.127 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:32:07.127 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:07.127 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.127 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.109, 2026-02-21T11:32:07.127 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.170, 2026-02-21T11:32:07.127 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.195 2026-02-21T11:32:07.127 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.127 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.127 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.105, 2026-02-21T11:32:07.127 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.093, 2026-02-21T11:32:07.127 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.093 2026-02-21T11:32:07.127 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.128 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.128 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.123, 2026-02-21T11:32:07.128 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.784, 2026-02-21T11:32:07.128 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.784 2026-02-21T11:32:07.128 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.128 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.140 2026-02-21T11:32:07.128 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.128 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.128 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:02 2026", 2026-02-21T11:32:07.128 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.128 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 1, 2026-02-21T11:32:07.128 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:32:07.128 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:07.129 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.129 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.099, 2026-02-21T11:32:07.129 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.176, 2026-02-21T11:32:07.129 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.178 2026-02-21T11:32:07.129 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.129 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.129 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.085, 2026-02-21T11:32:07.129 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.085, 2026-02-21T11:32:07.129 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.085 2026-02-21T11:32:07.129 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.129 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.129 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.123, 2026-02-21T11:32:07.129 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.611, 2026-02-21T11:32:07.130 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.611 2026-02-21T11:32:07.130 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.130 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.130 2026-02-21T11:32:07.130 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.130 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.130 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:03 2026", 2026-02-21T11:32:07.130 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.130 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:07.130 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:32:07.130 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:07.130 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.130 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.119, 2026-02-21T11:32:07.130 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.151, 2026-02-21T11:32:07.131 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.173 2026-02-21T11:32:07.131 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.131 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.131 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.096, 2026-02-21T11:32:07.131 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.096, 2026-02-21T11:32:07.131 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.096 2026-02-21T11:32:07.131 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.131 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.131 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.162, 2026-02-21T11:32:07.131 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.304, 2026-02-21T11:32:07.131 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.304 2026-02-21T11:32:07.131 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.131 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.161 2026-02-21T11:32:07.132 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.132 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.132 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:03 2026", 2026-02-21T11:32:07.132 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.132 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:07.132 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:32:07.132 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:07.132 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.132 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.111, 2026-02-21T11:32:07.132 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.148, 2026-02-21T11:32:07.132 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.162 2026-02-21T11:32:07.132 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.133 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.133 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.086, 2026-02-21T11:32:07.133 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.086, 2026-02-21T11:32:07.133 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.086 2026-02-21T11:32:07.133 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.133 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.133 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.142, 2026-02-21T11:32:07.133 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.269, 2026-02-21T11:32:07.133 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.269 2026-02-21T11:32:07.133 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.133 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.124 2026-02-21T11:32:07.133 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.133 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.134 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:02 2026", 2026-02-21T11:32:07.134 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.134 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 1, 2026-02-21T11:32:07.134 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:32:07.134 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:07.134 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.134 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.084, 2026-02-21T11:32:07.134 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.156, 2026-02-21T11:32:07.134 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.155 2026-02-21T11:32:07.134 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.134 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.135 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.076, 2026-02-21T11:32:07.135 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.076, 2026-02-21T11:32:07.135 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.076 2026-02-21T11:32:07.135 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.135 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.135 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.094, 2026-02-21T11:32:07.135 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.367, 2026-02-21T11:32:07.135 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.367 2026-02-21T11:32:07.135 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.135 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.138 2026-02-21T11:32:07.135 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.135 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.136 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:04 2026", 2026-02-21T11:32:07.136 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.136 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 1, 2026-02-21T11:32:07.136 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 0, 2026-02-21T11:32:07.136 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:07.136 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.136 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.093, 2026-02-21T11:32:07.136 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.146, 2026-02-21T11:32:07.136 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.138 2026-02-21T11:32:07.136 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.136 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.136 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.078, 2026-02-21T11:32:07.136 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.078, 2026-02-21T11:32:07.137 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.078 2026-02-21T11:32:07.137 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.137 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.137 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.109, 2026-02-21T11:32:07.137 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.362, 2026-02-21T11:32:07.137 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.362 2026-02-21T11:32:07.137 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.137 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.109 2026-02-21T11:32:07.137 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.137 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.137 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:01 2026", 2026-02-21T11:32:07.137 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.137 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 2, 2026-02-21T11:32:07.138 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 0, 2026-02-21T11:32:07.138 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:07.138 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.138 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.102, 2026-02-21T11:32:07.138 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.133, 2026-02-21T11:32:07.138 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.121 2026-02-21T11:32:07.138 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.138 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.138 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.081, 2026-02-21T11:32:07.138 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.081, 2026-02-21T11:32:07.138 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.081 2026-02-21T11:32:07.139 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.139 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.139 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.117, 2026-02-21T11:32:07.139 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.214, 2026-02-21T11:32:07.139 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.214 2026-02-21T11:32:07.139 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.139 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.140 2026-02-21T11:32:07.139 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.139 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.139 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:01 2026", 2026-02-21T11:32:07.139 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.139 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 2, 2026-02-21T11:32:07.140 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 0, 2026-02-21T11:32:07.140 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:07.140 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.140 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.094, 2026-02-21T11:32:07.140 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.132, 2026-02-21T11:32:07.140 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.118 2026-02-21T11:32:07.140 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.140 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.140 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.072, 2026-02-21T11:32:07.140 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.072, 2026-02-21T11:32:07.140 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.072 2026-02-21T11:32:07.140 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.140 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.140 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.121, 2026-02-21T11:32:07.141 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.280, 2026-02-21T11:32:07.141 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.280 2026-02-21T11:32:07.141 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.141 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.117 2026-02-21T11:32:07.141 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.141 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:07.141 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:04 2026", 2026-02-21T11:32:07.141 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:07.141 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 1, 2026-02-21T11:32:07.141 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 0, 2026-02-21T11:32:07.141 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:07.141 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:07.141 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.090, 2026-02-21T11:32:07.142 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.118, 2026-02-21T11:32:07.142 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.126 2026-02-21T11:32:07.142 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.142 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:07.142 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.070, 2026-02-21T11:32:07.142 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.065, 2026-02-21T11:32:07.142 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.065 2026-02-21T11:32:07.142 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.142 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:07.142 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.119, 2026-02-21T11:32:07.142 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.267, 2026-02-21T11:32:07.142 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.267 2026-02-21T11:32:07.142 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:07.143 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.119 2026-02-21T11:32:07.143 INFO:tasks.workunit.client.0.trial189.stdout: } 2026-02-21T11:32:07.143 INFO:tasks.workunit.client.0.trial189.stdout: ] 2026-02-21T11:32:07.143 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:32:07.144 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:07.144 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T11:32:07.144 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2026-02-21T11:32:07.144 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:07.144 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2026-02-21T11:32:07.145 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2026-02-21T11:32:07.145 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2026-02-21T11:32:27.130 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2026-02-21T11:32:27.131 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:32:27.131 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:32:27.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:32:27.355 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:32:27.355 INFO:tasks.workunit.client.0.trial189.stderr:2' 2026-02-21T11:32:27.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:32:27.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:32:27.355 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:32:27.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836496 2026-02-21T11:32:27.422 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836496 2026-02-21T11:32:27.422 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836496' 2026-02-21T11:32:27.422 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:32:27.422 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:32:27.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=42949672975 2026-02-21T11:32:27.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 42949672975 2026-02-21T11:32:27.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836496 1-42949672975' 2026-02-21T11:32:27.486 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:32:27.486 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-21T11:32:27.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=60129542159 2026-02-21T11:32:27.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 60129542159 2026-02-21T11:32:27.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836496 1-42949672975 2-60129542159' 2026-02-21T11:32:27.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:32:27.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836496 2026-02-21T11:32:27.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:32:27.551 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:32:27.551 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836496 2026-02-21T11:32:27.552 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:32:27.552 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836496 2026-02-21T11:32:27.553 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836496 2026-02-21T11:32:27.553 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2026-02-21T11:32:27.553 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:32:27.776 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836493 -lt 21474836496 2026-02-21T11:32:27.776 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:32:28.776 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:32:28.777 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:32:29.003 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836496 -lt 21474836496 2026-02-21T11:32:29.003 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:32:29.004 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-42949672975 2026-02-21T11:32:29.004 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:32:29.004 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:32:29.005 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-42949672975 2026-02-21T11:32:29.005 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:32:29.006 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=42949672975 2026-02-21T11:32:29.006 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 42949672975 2026-02-21T11:32:29.007 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2026-02-21T11:32:29.007 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:32:29.231 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 42949672975 -lt 42949672975 2026-02-21T11:32:29.231 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:32:29.231 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 2-60129542159 2026-02-21T11:32:29.231 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:32:29.233 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=2 2026-02-21T11:32:29.233 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 2-60129542159 2026-02-21T11:32:29.234 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:32:29.234 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=60129542159 2026-02-21T11:32:29.235 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.2 seq 60129542159 2026-02-21T11:32:29.235 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.2 seq 60129542159' 2026-02-21T11:32:29.235 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T11:32:29.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 60129542159 -lt 60129542159 2026-02-21T11:32:29.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T11:32:29.460 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2026-02-21T11:32:29.460 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:32:29.460 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:32:29.460 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:32:29.460 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:32:29.460 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:32:29.460 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-osd.0.asok 2026-02-21T11:32:29.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T11:32:29.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.47438/ceph-osd.0.asok dump_osd_network 2026-02-21T11:32:29.503 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:32:29.504 INFO:tasks.workunit.client.0.trial189.stdout: "threshold": 1000, 2026-02-21T11:32:29.504 INFO:tasks.workunit.client.0.trial189.stdout: "entries": [] 2026-02-21T11:32:29.504 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:32:29.508 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:29.508 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T11:32:29.509 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2026-02-21T11:32:29.510 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:29.511 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2026-02-21T11:32:29.512 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2026-02-21T11:32:29.514 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T11:32:29.514 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-21T11:32:29.514 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-21T11:32:29.514 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-21T11:32:29.514 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:32:29.514 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:32:29.515 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:32:29.515 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-mgr.x.asok 2026-02-21T11:32:29.515 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T11:32:29.515 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.47438/ceph-mgr.x.asok dump_osd_network 2026-02-21T11:32:29.552 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:32:29.552 INFO:tasks.workunit.client.0.trial189.stdout: "threshold": 1000, 2026-02-21T11:32:29.552 INFO:tasks.workunit.client.0.trial189.stdout: "entries": [] 2026-02-21T11:32:29.552 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:32:29.558 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:29.558 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T11:32:29.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2026-02-21T11:32:29.561 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:29.561 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2026-02-21T11:32:29.563 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2026-02-21T11:32:29.563 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T11:32:29.563 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2026-02-21T11:32:29.563 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:32:29.564 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:32:29.564 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:32:29.564 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:32:29.564 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:32:29.564 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-osd.0.asok 2026-02-21T11:32:29.565 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T11:32:29.565 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.47438/ceph-osd.0.asok dump_osd_network 0 2026-02-21T11:32:29.607 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:32:29.607 INFO:tasks.workunit.client.0.trial189.stdout: "threshold": 0, 2026-02-21T11:32:29.608 INFO:tasks.workunit.client.0.trial189.stdout: "entries": [ 2026-02-21T11:32:29.608 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.608 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:25 2026", 2026-02-21T11:32:29.608 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.608 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:29.608 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:32:29.608 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:29.608 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.608 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.119, 2026-02-21T11:32:29.608 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.123, 2026-02-21T11:32:29.608 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.236 2026-02-21T11:32:29.609 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.609 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.609 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.138, 2026-02-21T11:32:29.609 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.214, 2026-02-21T11:32:29.609 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 1.305 2026-02-21T11:32:29.609 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.609 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.609 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.138, 2026-02-21T11:32:29.609 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.214, 2026-02-21T11:32:29.609 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 1.305 2026-02-21T11:32:29.609 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.609 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.143 2026-02-21T11:32:29.610 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.610 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.610 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:25 2026", 2026-02-21T11:32:29.610 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.610 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:29.610 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:32:29.610 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:29.610 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.610 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.091, 2026-02-21T11:32:29.610 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.117, 2026-02-21T11:32:29.610 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.165 2026-02-21T11:32:29.610 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.611 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.611 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.097, 2026-02-21T11:32:29.611 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.223, 2026-02-21T11:32:29.611 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.784 2026-02-21T11:32:29.611 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.611 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.611 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.097, 2026-02-21T11:32:29.611 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.223, 2026-02-21T11:32:29.611 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.784 2026-02-21T11:32:29.611 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.611 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.125 2026-02-21T11:32:29.611 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.612 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.612 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:28 2026", 2026-02-21T11:32:29.612 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.612 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:29.612 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:32:29.612 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:29.612 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.612 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.131, 2026-02-21T11:32:29.612 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.124, 2026-02-21T11:32:29.612 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.153 2026-02-21T11:32:29.612 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.612 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.612 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.195, 2026-02-21T11:32:29.613 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.195, 2026-02-21T11:32:29.613 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.304 2026-02-21T11:32:29.613 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.613 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.613 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.195, 2026-02-21T11:32:29.613 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.195, 2026-02-21T11:32:29.613 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.304 2026-02-21T11:32:29.614 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.614 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.139 2026-02-21T11:32:29.614 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.614 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.614 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:28 2026", 2026-02-21T11:32:29.614 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.614 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:29.615 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:32:29.615 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:29.615 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.615 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.133, 2026-02-21T11:32:29.615 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.132, 2026-02-21T11:32:29.615 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.149 2026-02-21T11:32:29.615 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.616 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.616 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.205, 2026-02-21T11:32:29.616 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.205, 2026-02-21T11:32:29.616 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.269 2026-02-21T11:32:29.616 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.616 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.616 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.205, 2026-02-21T11:32:29.616 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.205, 2026-02-21T11:32:29.616 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.269 2026-02-21T11:32:29.617 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.617 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.154 2026-02-21T11:32:29.617 INFO:tasks.workunit.client.0.trial189.stdout: } 2026-02-21T11:32:29.617 INFO:tasks.workunit.client.0.trial189.stdout: ] 2026-02-21T11:32:29.617 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:32:29.619 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:29.619 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T11:32:29.619 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2026-02-21T11:32:29.619 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:29.619 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2026-02-21T11:32:29.619 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2026-02-21T11:32:29.620 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T11:32:29.620 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-21T11:32:29.620 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-21T11:32:29.620 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-21T11:32:29.620 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:32:29.620 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:32:29.620 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:32:29.620 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-mgr.x.asok 2026-02-21T11:32:29.620 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T11:32:29.620 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.47438/ceph-mgr.x.asok dump_osd_network 0 2026-02-21T11:32:29.657 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:32:29.657 INFO:tasks.workunit.client.0.trial189.stdout: "threshold": 0, 2026-02-21T11:32:29.657 INFO:tasks.workunit.client.0.trial189.stdout: "entries": [ 2026-02-21T11:32:29.657 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.657 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:25 2026", 2026-02-21T11:32:29.657 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.657 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:29.657 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:32:29.657 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:29.657 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.658 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.119, 2026-02-21T11:32:29.658 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.123, 2026-02-21T11:32:29.658 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.236 2026-02-21T11:32:29.658 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.658 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.658 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.101, 2026-02-21T11:32:29.658 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.080, 2026-02-21T11:32:29.658 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.080 2026-02-21T11:32:29.658 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.658 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.658 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.138, 2026-02-21T11:32:29.658 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.214, 2026-02-21T11:32:29.659 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 1.305 2026-02-21T11:32:29.659 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.659 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.106 2026-02-21T11:32:29.659 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.659 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.659 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:26 2026", 2026-02-21T11:32:29.659 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.659 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 2, 2026-02-21T11:32:29.659 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:32:29.659 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:29.659 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.659 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.136, 2026-02-21T11:32:29.660 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.136, 2026-02-21T11:32:29.660 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.175 2026-02-21T11:32:29.660 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.660 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.660 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.107, 2026-02-21T11:32:29.660 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.083, 2026-02-21T11:32:29.660 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.083 2026-02-21T11:32:29.660 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.660 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.660 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.229, 2026-02-21T11:32:29.661 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.234, 2026-02-21T11:32:29.661 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 1.520 2026-02-21T11:32:29.661 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.661 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.111 2026-02-21T11:32:29.661 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.661 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.661 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:26 2026", 2026-02-21T11:32:29.661 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.661 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 2, 2026-02-21T11:32:29.661 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:32:29.661 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:29.661 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.661 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.154, 2026-02-21T11:32:29.662 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.143, 2026-02-21T11:32:29.662 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.169 2026-02-21T11:32:29.662 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.662 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.662 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.114, 2026-02-21T11:32:29.662 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.090, 2026-02-21T11:32:29.662 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.090 2026-02-21T11:32:29.662 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.662 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.662 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.281, 2026-02-21T11:32:29.662 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.281, 2026-02-21T11:32:29.662 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 1.537 2026-02-21T11:32:29.662 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.663 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.119 2026-02-21T11:32:29.663 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.663 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.663 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:25 2026", 2026-02-21T11:32:29.663 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.663 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:29.663 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:32:29.663 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:29.663 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.663 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.091, 2026-02-21T11:32:29.663 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.117, 2026-02-21T11:32:29.663 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.165 2026-02-21T11:32:29.663 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.664 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.664 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.084, 2026-02-21T11:32:29.664 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.082, 2026-02-21T11:32:29.664 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.082 2026-02-21T11:32:29.664 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.664 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.664 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.097, 2026-02-21T11:32:29.664 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.223, 2026-02-21T11:32:29.664 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.784 2026-02-21T11:32:29.664 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.664 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.098 2026-02-21T11:32:29.664 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.665 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.665 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:27 2026", 2026-02-21T11:32:29.665 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.665 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 1, 2026-02-21T11:32:29.665 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:32:29.665 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:29.665 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.665 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.107, 2026-02-21T11:32:29.665 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.118, 2026-02-21T11:32:29.665 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.158 2026-02-21T11:32:29.665 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.665 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.665 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.093, 2026-02-21T11:32:29.666 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.089, 2026-02-21T11:32:29.666 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.085 2026-02-21T11:32:29.666 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.666 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.666 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.127, 2026-02-21T11:32:29.666 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.195, 2026-02-21T11:32:29.666 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.611 2026-02-21T11:32:29.666 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.666 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.104 2026-02-21T11:32:29.666 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.666 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.666 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:23 2026", 2026-02-21T11:32:29.666 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.667 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:29.667 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:32:29.667 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:29.667 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.667 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.104, 2026-02-21T11:32:29.667 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.121, 2026-02-21T11:32:29.667 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.157 2026-02-21T11:32:29.667 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.667 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.667 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.094, 2026-02-21T11:32:29.667 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.092, 2026-02-21T11:32:29.667 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.092 2026-02-21T11:32:29.668 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.668 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.668 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.111, 2026-02-21T11:32:29.668 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.192, 2026-02-21T11:32:29.668 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.304 2026-02-21T11:32:29.668 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.668 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.134 2026-02-21T11:32:29.668 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.668 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.668 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:23 2026", 2026-02-21T11:32:29.668 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.668 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:32:29.669 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:32:29.669 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:29.669 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.669 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.130, 2026-02-21T11:32:29.669 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.128, 2026-02-21T11:32:29.669 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.152 2026-02-21T11:32:29.669 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.669 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.669 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.111, 2026-02-21T11:32:29.669 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.076, 2026-02-21T11:32:29.669 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.076 2026-02-21T11:32:29.669 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.669 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.670 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.159, 2026-02-21T11:32:29.670 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.200, 2026-02-21T11:32:29.670 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.269 2026-02-21T11:32:29.670 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.670 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.126 2026-02-21T11:32:29.670 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.670 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.670 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:27 2026", 2026-02-21T11:32:29.670 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.670 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 1, 2026-02-21T11:32:29.670 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:32:29.670 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:29.670 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.671 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.092, 2026-02-21T11:32:29.671 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.106, 2026-02-21T11:32:29.671 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.139 2026-02-21T11:32:29.671 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.671 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.671 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.083, 2026-02-21T11:32:29.671 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.078, 2026-02-21T11:32:29.671 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.076 2026-02-21T11:32:29.671 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.671 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.671 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.101, 2026-02-21T11:32:29.671 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.186, 2026-02-21T11:32:29.671 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.367 2026-02-21T11:32:29.671 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.672 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.091 2026-02-21T11:32:29.672 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.672 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.672 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:24 2026", 2026-02-21T11:32:29.672 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.672 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 1, 2026-02-21T11:32:29.672 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 0, 2026-02-21T11:32:29.672 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:29.672 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.672 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.087, 2026-02-21T11:32:29.672 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.099, 2026-02-21T11:32:29.672 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.129 2026-02-21T11:32:29.672 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.673 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.673 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.076, 2026-02-21T11:32:29.673 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.074, 2026-02-21T11:32:29.673 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.074 2026-02-21T11:32:29.673 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.673 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.673 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.110, 2026-02-21T11:32:29.673 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.171, 2026-02-21T11:32:29.673 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.362 2026-02-21T11:32:29.673 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.673 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.070 2026-02-21T11:32:29.673 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.673 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.674 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:26 2026", 2026-02-21T11:32:29.674 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.674 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 2, 2026-02-21T11:32:29.674 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 0, 2026-02-21T11:32:29.674 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:32:29.674 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.674 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.117, 2026-02-21T11:32:29.674 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.115, 2026-02-21T11:32:29.674 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.121 2026-02-21T11:32:29.674 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.674 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.674 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.080, 2026-02-21T11:32:29.674 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.069, 2026-02-21T11:32:29.675 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.069 2026-02-21T11:32:29.675 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.675 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.675 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.217, 2026-02-21T11:32:29.675 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.217, 2026-02-21T11:32:29.675 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.217 2026-02-21T11:32:29.675 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.675 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.101 2026-02-21T11:32:29.675 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.675 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.675 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:26 2026", 2026-02-21T11:32:29.675 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.675 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 2, 2026-02-21T11:32:29.675 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 0, 2026-02-21T11:32:29.676 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:29.676 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.676 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.102, 2026-02-21T11:32:29.676 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.114, 2026-02-21T11:32:29.676 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.119 2026-02-21T11:32:29.676 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.676 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.676 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.063, 2026-02-21T11:32:29.676 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.063, 2026-02-21T11:32:29.676 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.063 2026-02-21T11:32:29.676 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.676 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.676 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.207, 2026-02-21T11:32:29.677 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.207, 2026-02-21T11:32:29.677 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.280 2026-02-21T11:32:29.677 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.677 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.081 2026-02-21T11:32:29.677 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.677 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:32:29.677 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:32:24 2026", 2026-02-21T11:32:29.677 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:32:29.677 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 1, 2026-02-21T11:32:29.677 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 0, 2026-02-21T11:32:29.677 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:32:29.677 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:32:29.677 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.085, 2026-02-21T11:32:29.677 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.089, 2026-02-21T11:32:29.678 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.115 2026-02-21T11:32:29.678 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.678 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:32:29.678 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.069, 2026-02-21T11:32:29.678 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.061, 2026-02-21T11:32:29.678 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.061 2026-02-21T11:32:29.678 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.678 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:32:29.678 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.107, 2026-02-21T11:32:29.678 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.162, 2026-02-21T11:32:29.678 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.267 2026-02-21T11:32:29.678 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:32:29.678 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.059 2026-02-21T11:32:29.679 INFO:tasks.workunit.client.0.trial189.stdout: } 2026-02-21T11:32:29.679 INFO:tasks.workunit.client.0.trial189.stdout: ] 2026-02-21T11:32:29.679 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:32:29.680 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:29.680 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T11:32:29.680 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2026-02-21T11:32:29.680 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:32:29.680 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2026-02-21T11:32:29.680 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2026-02-21T11:32:29.680 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2026-02-21T11:33:19.667 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2026-02-21T11:33:19.668 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:33:19.668 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:33:19.893 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:33:19.894 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:33:19.894 INFO:tasks.workunit.client.0.trial189.stderr:2' 2026-02-21T11:33:19.894 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:33:19.894 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:33:19.894 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:33:19.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836508 2026-02-21T11:33:19.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836508 2026-02-21T11:33:19.960 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836508' 2026-02-21T11:33:19.960 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:33:19.960 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:33:20.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=42949672988 2026-02-21T11:33:20.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 42949672988 2026-02-21T11:33:20.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836508 1-42949672988' 2026-02-21T11:33:20.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:33:20.024 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-21T11:33:20.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=60129542171 2026-02-21T11:33:20.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 60129542171 2026-02-21T11:33:20.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836508 1-42949672988 2-60129542171' 2026-02-21T11:33:20.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:33:20.091 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836508 2026-02-21T11:33:20.091 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:33:20.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:33:20.092 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836508 2026-02-21T11:33:20.092 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:33:20.093 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836508 2026-02-21T11:33:20.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2026-02-21T11:33:20.094 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836508 2026-02-21T11:33:20.095 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:33:20.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836506 -lt 21474836508 2026-02-21T11:33:20.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:33:21.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:33:21.317 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:33:21.541 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836506 -lt 21474836508 2026-02-21T11:33:21.542 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:33:22.542 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T11:33:22.543 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:33:22.768 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836508 -lt 21474836508 2026-02-21T11:33:22.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:33:22.769 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-42949672988 2026-02-21T11:33:22.769 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:33:22.770 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:33:22.771 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-42949672988 2026-02-21T11:33:22.771 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:33:22.772 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=42949672988 2026-02-21T11:33:22.772 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 42949672988 2026-02-21T11:33:22.772 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2026-02-21T11:33:22.772 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:33:22.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 42949672988 -lt 42949672988 2026-02-21T11:33:22.997 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:33:22.997 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 2-60129542171 2026-02-21T11:33:22.998 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:33:22.998 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=2 2026-02-21T11:33:22.999 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 2-60129542171 2026-02-21T11:33:22.999 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:33:23.000 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=60129542171 2026-02-21T11:33:23.000 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.2 seq 60129542171' 2026-02-21T11:33:23.000 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.2 seq 60129542171 2026-02-21T11:33:23.001 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T11:33:23.225 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 60129542172 -lt 60129542171 2026-02-21T11:33:23.226 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T11:33:23.227 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2026-02-21T11:33:23.227 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:33:23.228 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:33:23.228 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:33:23.228 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:23.228 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:23.228 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-osd.0.asok 2026-02-21T11:33:23.228 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T11:33:23.228 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.47438/ceph-osd.0.asok dump_osd_network 2026-02-21T11:33:23.271 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:33:23.271 INFO:tasks.workunit.client.0.trial189.stdout: "threshold": 1000, 2026-02-21T11:33:23.271 INFO:tasks.workunit.client.0.trial189.stdout: "entries": [] 2026-02-21T11:33:23.272 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:33:23.276 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:33:23.276 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T11:33:23.278 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2026-02-21T11:33:23.279 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:33:23.279 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2026-02-21T11:33:23.281 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2026-02-21T11:33:23.281 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T11:33:23.282 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-21T11:33:23.282 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-21T11:33:23.283 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-21T11:33:23.283 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:33:23.283 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:23.283 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:23.283 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-mgr.x.asok 2026-02-21T11:33:23.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T11:33:23.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.47438/ceph-mgr.x.asok dump_osd_network 2026-02-21T11:33:23.320 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:33:23.321 INFO:tasks.workunit.client.0.trial189.stdout: "threshold": 1000, 2026-02-21T11:33:23.321 INFO:tasks.workunit.client.0.trial189.stdout: "entries": [] 2026-02-21T11:33:23.321 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:33:23.325 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:33:23.325 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T11:33:23.327 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2026-02-21T11:33:23.328 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:33:23.328 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2026-02-21T11:33:23.329 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2026-02-21T11:33:23.330 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T11:33:23.331 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2026-02-21T11:33:23.331 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:33:23.331 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:33:23.331 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:33:23.331 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:23.331 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:23.332 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-osd.0.asok 2026-02-21T11:33:23.332 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T11:33:23.332 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.47438/ceph-osd.0.asok dump_osd_network 0 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "threshold": 0, 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "entries": [ 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:23 2026", 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.155, 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.117, 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.122 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.196, 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.196, 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.270 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.196, 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.196, 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.270 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.372 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.196 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:23 2026", 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.148, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.109, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.123 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.164, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.164, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.259 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.164, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.164, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.259 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.158 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:20 2026", 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.113, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.108, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.118 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.158, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.158, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.240 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.373 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.158, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.158, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.240 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.169 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:20 2026", 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.105, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.094, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.106 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.134, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.134, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.229 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.134, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.134, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.229 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.179 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: } 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout: ] 2026-02-21T11:33:23.374 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:33:23.376 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:33:23.376 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T11:33:23.378 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2026-02-21T11:33:23.379 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:33:23.379 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2026-02-21T11:33:23.381 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2026-02-21T11:33:23.381 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T11:33:23.382 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-21T11:33:23.382 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-21T11:33:23.382 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-21T11:33:23.382 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:33:23.382 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:23.382 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:23.382 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-mgr.x.asok 2026-02-21T11:33:23.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T11:33:23.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.47438/ceph-mgr.x.asok dump_osd_network 0 2026-02-21T11:33:23.419 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:33:23.420 INFO:tasks.workunit.client.0.trial189.stdout: "threshold": 0, 2026-02-21T11:33:23.420 INFO:tasks.workunit.client.0.trial189.stdout: "entries": [ 2026-02-21T11:33:23.420 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.420 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:21 2026", 2026-02-21T11:33:23.420 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.420 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 2, 2026-02-21T11:33:23.420 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:33:23.420 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:33:23.420 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.420 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.195, 2026-02-21T11:33:23.420 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.133, 2026-02-21T11:33:23.421 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.132 2026-02-21T11:33:23.421 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.421 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.421 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.098, 2026-02-21T11:33:23.421 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.078, 2026-02-21T11:33:23.421 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.078 2026-02-21T11:33:23.421 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.421 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.421 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.418, 2026-02-21T11:33:23.421 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.418, 2026-02-21T11:33:23.421 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.418 2026-02-21T11:33:23.421 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.421 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.418 2026-02-21T11:33:23.421 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.421 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.422 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:21 2026", 2026-02-21T11:33:23.422 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.422 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 2, 2026-02-21T11:33:23.422 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:33:23.422 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:33:23.422 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.422 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.185, 2026-02-21T11:33:23.422 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.121, 2026-02-21T11:33:23.422 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.125 2026-02-21T11:33:23.422 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.422 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.422 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.087, 2026-02-21T11:33:23.422 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.076, 2026-02-21T11:33:23.422 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.076 2026-02-21T11:33:23.422 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.423 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.423 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.376, 2026-02-21T11:33:23.423 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.376, 2026-02-21T11:33:23.423 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.376 2026-02-21T11:33:23.423 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.423 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.376 2026-02-21T11:33:23.423 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.423 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.423 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:21 2026", 2026-02-21T11:33:23.423 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.423 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 2, 2026-02-21T11:33:23.423 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 0, 2026-02-21T11:33:23.423 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:33:23.423 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.423 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.167, 2026-02-21T11:33:23.424 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.112, 2026-02-21T11:33:23.424 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.108 2026-02-21T11:33:23.424 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.424 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.424 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.093, 2026-02-21T11:33:23.424 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.074, 2026-02-21T11:33:23.424 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.069 2026-02-21T11:33:23.424 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.424 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.424 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.307, 2026-02-21T11:33:23.424 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.307, 2026-02-21T11:33:23.424 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.307 2026-02-21T11:33:23.424 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.424 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.307 2026-02-21T11:33:23.424 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:21 2026", 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 2, 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 0, 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.149, 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.108, 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.105 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.076, 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.070, 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.063 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.425 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.324, 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.324, 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.324 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.324 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:18 2026", 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.117, 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.108, 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.123 2026-02-21T11:33:23.426 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.427 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.427 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.070, 2026-02-21T11:33:23.427 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.070, 2026-02-21T11:33:23.427 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.070 2026-02-21T11:33:23.427 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.427 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.427 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.135, 2026-02-21T11:33:23.427 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.135, 2026-02-21T11:33:23.427 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.270 2026-02-21T11:33:23.427 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.427 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.104 2026-02-21T11:33:23.427 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.427 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.427 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:18 2026", 2026-02-21T11:33:23.427 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.101, 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.103, 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.123 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.083, 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.076, 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.076 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.122, 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.144, 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.259 2026-02-21T11:33:23.428 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.129 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:15 2026", 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.113, 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.114, 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.120 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.109, 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.087, 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.080 2026-02-21T11:33:23.429 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.124, 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.203, 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.240 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.113 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:17 2026", 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 1, 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.109, 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.112, 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.116 2026-02-21T11:33:23.430 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.086, 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.066, 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.066 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.136, 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.207, 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.207 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.109 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:15 2026", 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 0, 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 1, 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.102, 2026-02-21T11:33:23.431 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.101, 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.108 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.096, 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.073, 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.070 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.105, 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.209, 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.229 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.086 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:17 2026", 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 1, 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 2, 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:33:23.432 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.092, 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.095, 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.100 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.084, 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.073, 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.064 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.108, 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.221, 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.221 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.092 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:19 2026", 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 1, 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 0, 2026-02-21T11:33:23.433 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "front", 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.081, 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.094, 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.097 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.069, 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.065, 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.062 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.090, 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.160, 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.172 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.082 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: { 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "last update": "Sat Feb 21 11:33:19 2026", 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "stale": false, 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "from osd": 1, 2026-02-21T11:33:23.434 INFO:tasks.workunit.client.0.trial189.stdout: "to osd": 0, 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: "interface": "back", 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: "average": { 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.074, 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.070, 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.082 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: "min": { 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.058, 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.057, 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.056 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: "max": { 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: "1min": 0.091, 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: "5min": 0.091, 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: "15min": 0.180 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: }, 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: "last": 0.072 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: } 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout: ] 2026-02-21T11:33:23.435 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:33:23.436 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:33:23.437 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2026-02-21T11:33:23.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2026-02-21T11:33:23.437 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:33:23.437 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2026-02-21T11:33:23.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2026-02-21T11:33:23.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T11:33:23.437 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-21T11:33:23.437 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-21T11:33:23.437 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-21T11:33:23.437 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:33:23.437 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:23.437 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:23.437 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-mgr.x.asok 2026-02-21T11:33:23.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T11:33:23.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.47438/ceph-mgr.x.asok dump_osd_network 99 2026-02-21T11:33:23.467 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:33:23.467 INFO:tasks.workunit.client.0.trial189.stdout: "threshold": 99, 2026-02-21T11:33:23.467 INFO:tasks.workunit.client.0.trial189.stdout: "entries": [] 2026-02-21T11:33:23.467 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:33:23.471 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:33:23.471 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2026-02-21T11:33:23.473 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2026-02-21T11:33:23.473 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2026-02-21T11:33:23.475 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2026-02-21T11:33:23.475 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:33:23.475 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:33:23.475 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:33:23.475 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:23.475 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:23.475 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-osd.0.asok 2026-02-21T11:33:23.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2026-02-21T11:33:23.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.47438/ceph-osd.0.asok dump_osd_network 98 2026-02-21T11:33:23.518 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:33:23.518 INFO:tasks.workunit.client.0.trial189.stdout: "threshold": 98, 2026-02-21T11:33:23.518 INFO:tasks.workunit.client.0.trial189.stdout: "entries": [] 2026-02-21T11:33:23.518 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:33:23.523 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2026-02-21T11:33:23.523 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2026-02-21T11:33:23.525 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2026-02-21T11:33:23.525 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2026-02-21T11:33:23.525 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2026-02-21T11:33:23.526 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-21T11:33:23.526 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:33:23.526 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-21T11:33:23.527 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:33:23.527 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:33:23.527 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:33:23.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:33:23.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:33:23.634 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:33:23.634 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:33:23.635 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:33:23.635 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:33:23.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:33:23.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:33:23.636 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:33:23.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:33:23.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:33:23.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:33:23.638 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:33:23.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:33:23.639 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:33:23.639 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-21T11:33:24.264 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:33:24.264 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:24.264 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:24.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.47438 2026-02-21T11:33:24.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:33:24.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:33:24.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2026-02-21T11:33:24.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2026-02-21T11:33:24.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/network-ping 2026-02-21T11:33:24.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/network-ping 2026-02-21T11:33:24.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-21T11:33:24.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:33:24.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-21T11:33:24.268 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:33:24.268 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:33:24.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:33:24.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:33:24.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:33:24.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:33:24.268 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:33:24.269 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:33:24.269 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:33:24.269 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:33:24.269 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:33:24.269 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:33:24.270 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:33:24.270 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:33:24.270 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:33:24.270 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:33:24.271 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:33:24.272 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:33:24.272 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-21T11:33:24.273 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:33:24.273 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:24.273 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:24.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.47438 2026-02-21T11:33:24.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:33:24.275 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:33:24.275 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/network-ping 2026-02-21T11:33:24.276 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:33:24.276 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:24.276 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:24.276 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.47438 2026-02-21T11:33:24.278 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:33:24.278 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:33:24.278 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:33:24.278 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2026-02-21T11:33:24.278 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2026-02-21T11:33:24.278 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2026-02-21T11:33:24.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2026-02-21T11:33:24.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2026-02-21T11:33:24.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2026-02-21T11:33:24.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2026-02-21T11:33:24.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/network-ping 2026-02-21T11:33:24.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:33:24.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:33:24.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:33:24.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/network-ping/a 2026-02-21T11:33:24.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2026-02-21T11:33:24.297 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:33:24.297 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:33:24.297 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:33:24.297 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:33:24.298 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:24.298 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:24.298 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:33:24.298 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T11:33:24.321 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:33:24.322 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:33:24.322 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:33:24.322 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:33:24.322 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:33:24.322 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:33:24.323 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:33:24.323 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:33:24.324 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:33:24.324 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:33:24.324 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:24.324 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:24.324 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-mon.a.asok 2026-02-21T11:33:24.324 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:33:24.324 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.47438/ceph-mon.a.asok config get fsid 2026-02-21T11:33:24.366 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:33:24.366 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:33:24.366 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:33:24.366 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:33:24.367 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:33:24.367 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:33:24.367 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:33:24.367 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:33:24.367 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:33:24.367 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:24.367 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:24.367 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.47438/ceph-mon.a.asok 2026-02-21T11:33:24.367 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:33:24.367 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.47438/ceph-mon.a.asok config get mon_host 2026-02-21T11:33:24.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2026-02-21T11:33:24.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/network-ping 2026-02-21T11:33:24.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:33:24.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:33:24.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:33:24.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/network-ping/x 2026-02-21T11:33:24.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:33:24.507 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:33:24.507 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:33:24.507 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:33:24.507 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:33:24.507 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:24.508 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:24.508 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:33:24.508 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:33:24.509 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:33:24.527 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2026-02-21T11:33:24.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-21T11:33:24.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:33:24.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:33:24.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:33:24.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/0 2026-02-21T11:33:24.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-21T11:33:24.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:33:24.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:33:24.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:33:24.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2026-02-21T11:33:24.529 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2026-02-21T11:33:24.529 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:33:24.529 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2026-02-21T11:33:24.529 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-21T11:33:24.529 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:33:24.529 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:33:24.529 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:33:24.529 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:33:24.529 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:24.529 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:24.529 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:33:24.529 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:33:24.530 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:33:24.530 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:33:24.530 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:33:24.530 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-21T11:33:24.530 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-21T11:33:24.530 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:33:24.530 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:33:24.530 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:33:24.530 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:33:24.530 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:33:24.530 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:33:24.530 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/0 2026-02-21T11:33:24.531 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:33:24.531 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d0c18b4a-dfb4-4818-9170-28ee40f6c4eb 2026-02-21T11:33:24.531 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d0c18b4a-dfb4-4818-9170-28ee40f6c4eb' 2026-02-21T11:33:24.532 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 d0c18b4a-dfb4-4818-9170-28ee40f6c4eb 2026-02-21T11:33:24.532 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:33:24.538 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAEmJlpBEwHIBAAvktS+/JRWyDFnPHmXsDd2Q== 2026-02-21T11:33:24.539 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAEmJlpBEwHIBAAvktS+/JRWyDFnPHmXsDd2Q=="}' 2026-02-21T11:33:24.539 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d0c18b4a-dfb4-4818-9170-28ee40f6c4eb -i td/network-ping/0/new.json 2026-02-21T11:33:24.642 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:33:24.649 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/0/new.json 2026-02-21T11:33:24.650 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAEmJlpBEwHIBAAvktS+/JRWyDFnPHmXsDd2Q== --osd-uuid d0c18b4a-dfb4-4818-9170-28ee40f6c4eb 2026-02-21T11:33:24.667 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:24.665+0000 7fc7a5c3b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:24.668 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:24.667+0000 7fc7a5c3b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:24.669 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:24.668+0000 7fc7a5c3b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:24.670 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:24.668+0000 7fc7a5c3b780 -1 bdev(0x28a0c00 td/network-ping/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:33:24.670 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:24.668+0000 7fc7a5c3b780 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2026-02-21T11:33:25.081 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/0/keyring 2026-02-21T11:33:25.081 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:33:25.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:33:25.082 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:33:25.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:33:25.186 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:33:25.187 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:33:25.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:33:25.188 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:33:25.188 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:33:25.188 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:33:25.201 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:25.199+0000 7febe9c19780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:25.202 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:25.200+0000 7febe9c19780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:25.203 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:25.201+0000 7febe9c19780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:25.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:33:25.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:33:25.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:33:25.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:33:25.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:33:25.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:33:25.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:33:25.283 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:33:25.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:33:25.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:33:25.381 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:33:25.413 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:25.411+0000 7febe9c19780 -1 Falling back to public interface 2026-02-21T11:33:25.781 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:25.779+0000 7febe9c19780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:33:26.383 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:33:26.383 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:33:26.383 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:33:26.384 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:33:26.384 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:33:26.384 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:33:26.478 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1169901540,v1:127.0.0.1:6801/1169901540] [v2:127.0.0.1:6802/1169901540,v1:127.0.0.1:6803/1169901540] exists,up d0c18b4a-dfb4-4818-9170-28ee40f6c4eb 2026-02-21T11:33:26.482 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:33:26.482 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:33:26.482 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:33:26.482 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2026-02-21T11:33:26.482 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-21T11:33:26.482 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:33:26.482 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T11:33:26.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:33:26.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/1 2026-02-21T11:33:26.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-21T11:33:26.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:33:26.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:33:26.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:33:26.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2026-02-21T11:33:26.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2026-02-21T11:33:26.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:33:26.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2026-02-21T11:33:26.484 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-21T11:33:26.484 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:33:26.484 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:33:26.484 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:33:26.484 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:33:26.484 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:26.484 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:26.484 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:33:26.484 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:33:26.484 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:33:26.484 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:33:26.484 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:33:26.484 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-21T11:33:26.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-21T11:33:26.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:33:26.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:33:26.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:33:26.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:33:26.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:33:26.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:33:26.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/1 2026-02-21T11:33:26.485 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:33:26.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2b88febb-2947-45dd-a6ed-ee4b814a0e00 2026-02-21T11:33:26.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 2b88febb-2947-45dd-a6ed-ee4b814a0e00' 2026-02-21T11:33:26.486 INFO:tasks.workunit.client.0.trial189.stdout:add osd1 2b88febb-2947-45dd-a6ed-ee4b814a0e00 2026-02-21T11:33:26.486 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:33:26.492 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAGmJlpEOY9HRAAuJ5/CaxCpWeNWccYEKW/Zg== 2026-02-21T11:33:26.492 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAGmJlpEOY9HRAAuJ5/CaxCpWeNWccYEKW/Zg=="}' 2026-02-21T11:33:26.492 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2b88febb-2947-45dd-a6ed-ee4b814a0e00 -i td/network-ping/1/new.json 2026-02-21T11:33:26.593 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:33:26.598 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/1/new.json 2026-02-21T11:33:26.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAGmJlpEOY9HRAAuJ5/CaxCpWeNWccYEKW/Zg== --osd-uuid 2b88febb-2947-45dd-a6ed-ee4b814a0e00 2026-02-21T11:33:26.615 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:26.614+0000 7feb9d434780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:26.617 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:26.615+0000 7feb9d434780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:26.618 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:26.616+0000 7feb9d434780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:26.618 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:26.616+0000 7feb9d434780 -1 bdev(0x3c66c00 td/network-ping/1/block) open stat got: (1) Operation not permitted 2026-02-21T11:33:26.618 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:26.616+0000 7feb9d434780 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2026-02-21T11:33:26.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/1/keyring 2026-02-21T11:33:26.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:33:26.934 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T11:33:26.934 INFO:tasks.workunit.client.0.trial189.stdout:adding osd1 key to auth repository 2026-02-21T11:33:26.934 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:33:27.251 INFO:tasks.workunit.client.0.trial189.stdout:start osd.1 2026-02-21T11:33:27.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T11:33:27.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:33:27.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:33:27.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:33:27.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:33:27.266 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:27.264+0000 7f016f659780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:27.268 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:27.266+0000 7f016f659780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:27.268 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:27.266+0000 7f016f659780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:27.400 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:27.399+0000 7f016f659780 -1 Falling back to public interface 2026-02-21T11:33:27.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T11:33:27.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:33:27.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T11:33:27.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:33:27.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:33:27.479 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:33:27.479 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:33:27.479 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:33:27.479 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:33:27.480 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:33:27.699 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:33:27.706 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:27.704+0000 7f016f659780 -1 osd.1 0 log_to_monitors true 2026-02-21T11:33:28.701 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:33:28.701 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:33:28.701 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:33:28.701 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:33:28.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:33:28.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:33:28.924 INFO:tasks.workunit.client.0.trial189.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2131227969,v1:127.0.0.1:6811/2131227969] [v2:127.0.0.1:6812/2131227969,v1:127.0.0.1:6813/2131227969] exists,up 2b88febb-2947-45dd-a6ed-ee4b814a0e00 2026-02-21T11:33:28.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:33:28.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:33:28.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:33:28.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2026-02-21T11:33:28.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-21T11:33:28.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:33:28.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T11:33:28.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:33:28.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/2 2026-02-21T11:33:28.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-21T11:33:28.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:33:28.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:33:28.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:33:28.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2026-02-21T11:33:28.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2026-02-21T11:33:28.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:33:28.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2026-02-21T11:33:28.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-21T11:33:28.926 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:33:28.926 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:33:28.926 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:33:28.926 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:33:28.926 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:33:28.926 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:33:28.926 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:33:28.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' 2026-02-21T11:33:28.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:33:28.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:33:28.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:33:28.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-21T11:33:28.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-21T11:33:28.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:33:28.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:33:28.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:33:28.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:33:28.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:33:28.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:33:28.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/2 2026-02-21T11:33:28.927 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:33:28.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c03143cc-8cce-4743-916d-eacbbf17161e 2026-02-21T11:33:28.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 c03143cc-8cce-4743-916d-eacbbf17161e' 2026-02-21T11:33:28.928 INFO:tasks.workunit.client.0.trial189.stdout:add osd2 c03143cc-8cce-4743-916d-eacbbf17161e 2026-02-21T11:33:28.928 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:33:28.938 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAImJlpzhDeNxAAoEBNuvwTP0TBYPWvbBhw2Q== 2026-02-21T11:33:28.938 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAImJlpzhDeNxAAoEBNuvwTP0TBYPWvbBhw2Q=="}' 2026-02-21T11:33:28.938 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c03143cc-8cce-4743-916d-eacbbf17161e -i td/network-ping/2/new.json 2026-02-21T11:33:29.164 INFO:tasks.workunit.client.0.trial189.stdout:2 2026-02-21T11:33:29.171 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/2/new.json 2026-02-21T11:33:29.172 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAImJlpzhDeNxAAoEBNuvwTP0TBYPWvbBhw2Q== --osd-uuid c03143cc-8cce-4743-916d-eacbbf17161e 2026-02-21T11:33:29.189 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:29.187+0000 7fd73263c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:29.190 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:29.188+0000 7fd73263c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:29.191 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:29.189+0000 7fd73263c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:29.192 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:29.190+0000 7fd73263c780 -1 bdev(0x2c54c00 td/network-ping/2/block) open stat got: (1) Operation not permitted 2026-02-21T11:33:29.192 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:29.190+0000 7fd73263c780 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2026-02-21T11:33:29.526 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/2/keyring 2026-02-21T11:33:29.526 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:33:29.526 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T11:33:29.527 INFO:tasks.workunit.client.0.trial189.stdout:adding osd2 key to auth repository 2026-02-21T11:33:29.527 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:33:29.841 INFO:tasks.workunit.client.0.trial189.stdout:start osd.2 2026-02-21T11:33:29.841 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T11:33:29.841 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f67381a4-36f7-4f7e-bf9d-92fd94d0f5ec --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.47438/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:33:29.841 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:33:29.841 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:33:29.841 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:33:29.856 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:29.854+0000 7f53a1644780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:29.857 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:29.856+0000 7f53a1644780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:29.858 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:29.857+0000 7f53a1644780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:33:29.990 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:29.988+0000 7f53a1644780 -1 Falling back to public interface 2026-02-21T11:33:30.065 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T11:33:30.065 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:33:30.065 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T11:33:30.065 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:33:30.065 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:33:30.065 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:33:30.065 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:33:30.066 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:33:30.066 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:33:30.066 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T11:33:30.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:33:30.312 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:33:30.309+0000 7f53a1644780 -1 osd.2 0 log_to_monitors true 2026-02-21T11:33:31.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:33:31.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:33:31.291 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:33:31.292 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:33:31.292 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:33:31.292 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T11:33:31.517 INFO:tasks.workunit.client.0.trial189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1201666376,v1:127.0.0.1:6819/1201666376] [v2:127.0.0.1:6820/1201666376,v1:127.0.0.1:6821/1201666376] exists,up c03143cc-8cce-4743-916d-eacbbf17161e 2026-02-21T11:33:31.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:33:31.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:33:31.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:33:31.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2026-02-21T11:33:36.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2026-02-21T11:33:36.806 INFO:tasks.workunit.client.0.trial189.stderr:bucket 'dc1' already exists 2026-02-21T11:33:36.814 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2026-02-21T11:33:37.093 INFO:tasks.workunit.client.0.trial189.stderr:bucket 'dc2' already exists 2026-02-21T11:33:37.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2026-02-21T11:33:37.373 INFO:tasks.workunit.client.0.trial189.stderr:bucket 'dc3' already exists 2026-02-21T11:33:37.380 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2026-02-21T11:33:37.650 INFO:tasks.workunit.client.0.trial189.stderr:bucket 'rack1' already exists 2026-02-21T11:33:37.657 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2026-02-21T11:33:37.928 INFO:tasks.workunit.client.0.trial189.stderr:bucket 'rack2' already exists 2026-02-21T11:33:37.936 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2026-02-21T11:33:38.206 INFO:tasks.workunit.client.0.trial189.stderr:bucket 'rack3' already exists 2026-02-21T11:33:38.213 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2026-02-21T11:33:38.485 INFO:tasks.workunit.client.0.trial189.stderr:bucket 'host1' already exists 2026-02-21T11:33:38.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2026-02-21T11:33:38.791 INFO:tasks.workunit.client.0.trial189.stderr:bucket 'host2' already exists 2026-02-21T11:33:38.799 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2026-02-21T11:33:39.073 INFO:tasks.workunit.client.0.trial189.stderr:bucket 'host3' already exists 2026-02-21T11:33:39.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2026-02-21T11:33:39.352 INFO:tasks.workunit.client.0.trial189.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2026-02-21T11:33:39.360 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2026-02-21T11:33:39.636 INFO:tasks.workunit.client.0.trial189.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2026-02-21T11:33:39.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2026-02-21T11:33:39.914 INFO:tasks.workunit.client.0.trial189.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2026-02-21T11:33:39.921 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2026-02-21T11:33:40.198 INFO:tasks.workunit.client.0.trial189.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2026-02-21T11:33:40.205 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2026-02-21T11:33:40.477 INFO:tasks.workunit.client.0.trial189.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2026-02-21T11:33:40.484 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2026-02-21T11:33:40.782 INFO:tasks.workunit.client.0.trial189.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2026-02-21T11:33:40.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2026-02-21T11:33:41.062 INFO:tasks.workunit.client.0.trial189.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2026-02-21T11:33:41.070 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2026-02-21T11:33:41.342 INFO:tasks.workunit.client.0.trial189.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2026-02-21T11:33:41.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2026-02-21T11:33:41.623 INFO:tasks.workunit.client.0.trial189.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2026-02-21T11:33:41.631 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2026-02-21T11:33:41.902 INFO:tasks.workunit.client.0.trial189.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2026-02-21T11:33:41.910 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2026-02-21T11:33:42.186 INFO:tasks.workunit.client.0.trial189.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2026-02-21T11:33:42.193 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2026-02-21T11:33:42.467 INFO:tasks.workunit.client.0.trial189.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2026-02-21T11:33:42.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2026-02-21T11:33:42.772 INFO:tasks.workunit.client.0.trial189.stderr:rule myrule already exists 2026-02-21T11:33:42.779 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2026-02-21T11:33:42.779 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 16 replicated myrule 2026-02-21T11:33:43.058 INFO:tasks.workunit.client.0.trial189.stderr:pool 'foo' already exists 2026-02-21T11:33:43.067 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:33:44.068 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2026-02-21T11:33:44.089 INFO:tasks.workunit.client.0.trial189.stdout:hints = 1 2026-02-21T11:33:44.089 INFO:tasks.workunit.client.0.trial189.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2026-02-21T11:33:44.089 INFO:tasks.workunit.client.0.trial189.stdout:Object prefix: benchmark_data_trial189_57683 2026-02-21T11:33:44.089 INFO:tasks.workunit.client.0.trial189.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-21T11:33:44.090 INFO:tasks.workunit.client.0.trial189.stdout: 0 0 0 0 0 0 - 0 2026-02-21T11:33:45.088 INFO:tasks.workunit.client.0.trial189.stdout: 1 16 11562 11546 45.098 45.1016 0.00168534 0.00138344 2026-02-21T11:33:46.088 INFO:tasks.workunit.client.0.trial189.stdout: 2 16 24143 24127 47.1193 49.1445 0.00395241 0.00132402 2026-02-21T11:33:47.088 INFO:tasks.workunit.client.0.trial189.stdout: 3 16 36358 36342 47.3165 47.7148 0.00114992 0.00131936 2026-02-21T11:33:48.089 INFO:tasks.workunit.client.0.trial189.stdout: 4 16 48990 48974 47.8224 49.3438 0.00125537 0.00130551 2026-02-21T11:33:49.089 INFO:tasks.workunit.client.0.trial189.stdout: 5 16 61268 61252 47.8499 47.9609 0.00106357 0.00130478 2026-02-21T11:33:50.088 INFO:tasks.workunit.client.0.trial189.stdout: 6 16 73513 73497 47.8463 47.832 0.000948633 0.00130489 2026-02-21T11:33:51.089 INFO:tasks.workunit.client.0.trial189.stdout: 7 16 85202 85186 47.5334 45.6602 0.00131386 0.00131353 2026-02-21T11:33:52.089 INFO:tasks.workunit.client.0.trial189.stdout: 8 16 96049 96033 46.8877 42.3711 0.000968461 0.00133163 2026-02-21T11:33:53.089 INFO:tasks.workunit.client.0.trial189.stdout: 9 16 106830 106814 46.3569 42.1133 0.00137987 0.0013469 2026-02-21T11:33:54.089 INFO:tasks.workunit.client.0.trial189.stdout: 10 6 117430 117424 45.8654 41.4453 0.00139999 0.00136135 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Total time run: 10.001 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Total writes made: 117430 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Write size: 4096 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Object size: 4096 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Bandwidth (MB/sec): 45.8666 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Stddev Bandwidth: 2.99967 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Max bandwidth (MB/sec): 49.3438 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Min bandwidth (MB/sec): 41.4453 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Average IOPS: 11741 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Stddev IOPS: 767.916 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Max IOPS: 12632 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Min IOPS: 10610 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Average Latency(s): 0.00136134 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Stddev Latency(s): 0.000648496 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Max latency(s): 0.0114098 2026-02-21T11:33:55.089 INFO:tasks.workunit.client.0.trial189.stdout:Min latency(s): 0.000415327 2026-02-21T11:33:55.093 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2026-02-21T11:34:05.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2026-02-21T11:34:05.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:34:05.096 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:34:05.323 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:34:05.323 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:34:05.323 INFO:tasks.workunit.client.0.trial189.stderr:2' 2026-02-21T11:34:05.323 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:34:05.323 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:05.323 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:34:05.388 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836490 2026-02-21T11:34:05.388 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836490 2026-02-21T11:34:05.388 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836490' 2026-02-21T11:34:05.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:05.389 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:34:05.451 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=38654705674 2026-02-21T11:34:05.452 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 38654705674 2026-02-21T11:34:05.452 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836490 1-38654705674' 2026-02-21T11:34:05.452 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:05.452 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-21T11:34:05.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=60129542153 2026-02-21T11:34:05.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 60129542153 2026-02-21T11:34:05.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836490 1-38654705674 2-60129542153' 2026-02-21T11:34:05.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:05.517 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836490 2026-02-21T11:34:05.517 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:05.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:34:05.518 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836490 2026-02-21T11:34:05.518 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:05.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836490 2026-02-21T11:34:05.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-21T11:34:05.519 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836490 2026-02-21T11:34:05.520 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:34:05.742 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836488 -lt 21474836490 2026-02-21T11:34:05.742 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:34:06.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:34:06.743 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:34:06.967 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836490 -lt 21474836490 2026-02-21T11:34:06.968 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:06.968 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-38654705674 2026-02-21T11:34:06.968 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:06.969 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:34:06.969 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-38654705674 2026-02-21T11:34:06.969 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:06.970 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=38654705674 2026-02-21T11:34:06.970 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 38654705674' 2026-02-21T11:34:06.970 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 38654705674 2026-02-21T11:34:06.971 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:34:07.194 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 38654705674 -lt 38654705674 2026-02-21T11:34:07.194 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:07.194 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 2-60129542153 2026-02-21T11:34:07.194 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:07.195 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=2 2026-02-21T11:34:07.196 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 2-60129542153 2026-02-21T11:34:07.196 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:07.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=60129542153 2026-02-21T11:34:07.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-21T11:34:07.197 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.2 seq 60129542153 2026-02-21T11:34:07.197 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T11:34:07.427 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 60129542154 -lt 60129542153 2026-02-21T11:34:07.428 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2026-02-21T11:34:07.428 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2026-02-21T11:34:07.705 INFO:tasks.workunit.client.0.trial189.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.480ms); Slow OSD heartbeats on front (longest 0.477ms) 2026-02-21T11:34:07.712 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2026-02-21T11:34:07.713 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2026-02-21T11:34:07.713 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2026-02-21T11:34:08.018 INFO:tasks.workunit.client.0.trial189.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.480ms); Slow OSD heartbeats on front (longest 0.477ms) 2026-02-21T11:34:08.019 INFO:tasks.workunit.client.0.trial189.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.480ms) 2026-02-21T11:34:08.019 INFO:tasks.workunit.client.0.trial189.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.480 msec possibly improving 2026-02-21T11:34:08.019 INFO:tasks.workunit.client.0.trial189.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.202 msec 2026-02-21T11:34:08.019 INFO:tasks.workunit.client.0.trial189.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.169 msec 2026-02-21T11:34:08.019 INFO:tasks.workunit.client.0.trial189.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.165 msec 2026-02-21T11:34:08.019 INFO:tasks.workunit.client.0.trial189.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.156 msec 2026-02-21T11:34:08.019 INFO:tasks.workunit.client.0.trial189.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.125 msec 2026-02-21T11:34:08.019 INFO:tasks.workunit.client.0.trial189.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.477ms) 2026-02-21T11:34:08.019 INFO:tasks.workunit.client.0.trial189.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.477 msec possibly improving 2026-02-21T11:34:08.019 INFO:tasks.workunit.client.0.trial189.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.237 msec 2026-02-21T11:34:08.019 INFO:tasks.workunit.client.0.trial189.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.173 msec 2026-02-21T11:34:08.019 INFO:tasks.workunit.client.0.trial189.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.168 msec 2026-02-21T11:34:08.020 INFO:tasks.workunit.client.0.trial189.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.149 msec 2026-02-21T11:34:08.020 INFO:tasks.workunit.client.0.trial189.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.148 msec 2026-02-21T11:34:08.026 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2026-02-21T11:34:08.027 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2026-02-21T11:34:08.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2026-02-21T11:34:08.029 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2026-02-21T11:34:08.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2026-02-21T11:34:08.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-21T11:34:08.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:34:08.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-21T11:34:08.031 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:08.031 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:08.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:08.032 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:08.032 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:08.138 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:08.138 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:34:08.139 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:34:08.139 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:34:08.140 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:34:08.140 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:34:08.140 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:34:08.141 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:08.141 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:34:08.141 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:34:08.142 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:08.142 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:34:08.143 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:34:08.143 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-21T11:34:08.784 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:34:08.785 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:08.785 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:34:08.785 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.47438 2026-02-21T11:34:08.786 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:34:08.786 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:34:08.787 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: main: code=0 2026-02-21T11:34:08.787 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2433: main: teardown td/network-ping 0 2026-02-21T11:34:08.787 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-21T11:34:08.787 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-21T11:34:08.787 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-21T11:34:08.787 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:08.787 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:08.787 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:08.787 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:08.787 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:08.788 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:08.788 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:34:08.789 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:34:08.789 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:34:08.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:34:08.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:34:08.790 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:34:08.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:08.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:34:08.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:34:08.792 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:08.792 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:34:08.793 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-21T11:34:08.793 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-21T11:34:08.794 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:34:08.794 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:08.794 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.47438 2026-02-21T11:34:08.795 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.47438 2026-02-21T11:34:08.795 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:34:08.795 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:34:08.795 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: main: return 0 2026-02-21T11:34:08.797 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-21T11:34:08.797 DEBUG:teuthology.orchestra.run.trial189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-21T11:34:08.864 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2026-02-21T11:34:08.865 DEBUG:teuthology.orchestra.run.trial189:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2026-02-21T11:34:08.926 INFO:tasks.workunit.client.0.trial189.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-21T11:34:08.929 INFO:tasks.workunit.client.0.trial189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-21T11:34:08.929 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-21T11:34:08.929 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-21T11:34:08.929 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: main: export PYTHONWARNINGS=ignore 2026-02-21T11:34:08.929 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: main: PYTHONWARNINGS=ignore 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: main: export CEPH_CONF=/dev/null 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: main: CEPH_CONF=/dev/null 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: main: unset CEPH_ARGS 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: main: local code 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: main: run td/ok-to-stop 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2026-02-21T11:34:08.930 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2026-02-21T11:34:08.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none ' 2026-02-21T11:34:08.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none ' 2026-02-21T11:34:08.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none ' 2026-02-21T11:34:08.931 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2026-02-21T11:34:08.931 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-21T11:34:08.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2026-02-21T11:34:08.932 INFO:tasks.workunit.client.0.trial189.stderr:TEST_0_osd 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr:TEST_1_mon_checks 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr:TEST_2_mons_checks 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr:TEST_3_mons_checks 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr:TEST_4_mons_checks 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr:TEST_5_mons_checks' 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:08.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:08.934 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:08.934 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:34:08.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:34:08.935 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:34:08.936 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:34:08.936 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:34:08.936 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:34:08.937 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:08.937 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:34:08.937 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:34:08.937 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:08.938 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:34:08.939 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:34:08.939 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T11:34:08.940 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:34:08.940 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:08.940 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:08.940 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.58069 2026-02-21T11:34:08.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:34:08.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:34:08.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-21T11:34:08.942 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:34:08.942 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:08.942 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:08.942 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.58069 2026-02-21T11:34:08.943 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:34:08.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-21T11:34:08.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-21T11:34:08.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:34:08.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-21T11:34:08.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2026-02-21T11:34:08.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2026-02-21T11:34:08.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T11:34:08.945 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-21T11:34:08.945 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:34:08.945 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:34:08.945 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:34:08.945 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:34:08.945 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-21T11:34:08.945 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-21T11:34:08.962 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:34:08.962 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:08.962 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:08.963 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:08.963 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:08.963 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:08.963 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:08.963 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-21T11:34:08.985 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:34:08.987 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:34:08.987 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:08.987 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:34:08.988 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:34:08.988 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:34:08.988 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:34:08.988 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:34:08.988 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:34:08.988 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:08.988 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:08.988 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:08.988 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.a.asok 2026-02-21T11:34:08.988 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:08.988 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.a.asok config get fsid 2026-02-21T11:34:09.029 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:34:09.029 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:09.029 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:34:09.029 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:34:09.029 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:34:09.029 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:34:09.029 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:34:09.029 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:34:09.029 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:09.029 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:09.029 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:09.030 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.a.asok 2026-02-21T11:34:09.030 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:09.030 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.a.asok config get mon_host 2026-02-21T11:34:09.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2026-02-21T11:34:09.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-stop 2026-02-21T11:34:09.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:34:09.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:34:09.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:34:09.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-stop/x 2026-02-21T11:34:09.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:34:09.169 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:34:09.169 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:09.169 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:09.169 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:09.169 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:09.169 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:09.169 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:09.169 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:34:09.170 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:34:09.186 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2026-02-21T11:34:09.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-21T11:34:09.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:34:09.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:34:09.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:34:09.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/0 2026-02-21T11:34:09.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T11:34:09.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:34:09.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:34:09.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:34:09.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2026-02-21T11:34:09.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2026-02-21T11:34:09.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:34:09.188 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:34:09.188 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-21T11:34:09.188 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:34:09.188 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:09.188 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:09.188 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:09.188 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:09.188 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:09.188 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:09.188 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:09.188 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:34:09.188 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:34:09.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:34:09.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-21T11:34:09.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-21T11:34:09.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:34:09.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:34:09.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:34:09.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:34:09.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:34:09.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:34:09.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/0 2026-02-21T11:34:09.189 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:34:09.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ca954ccf-6dcf-4629-a0b6-53289c25af93 2026-02-21T11:34:09.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ca954ccf-6dcf-4629-a0b6-53289c25af93' 2026-02-21T11:34:09.190 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 ca954ccf-6dcf-4629-a0b6-53289c25af93 2026-02-21T11:34:09.190 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:34:09.195 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAxmJlpR7mMCxAAYMkzgQwAx5NM3nXzQUFdxA== 2026-02-21T11:34:09.195 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAxmJlpR7mMCxAAYMkzgQwAx5NM3nXzQUFdxA=="}' 2026-02-21T11:34:09.195 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ca954ccf-6dcf-4629-a0b6-53289c25af93 -i td/ok-to-stop/0/new.json 2026-02-21T11:34:09.294 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:34:09.300 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/0/new.json 2026-02-21T11:34:09.301 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxmJlpR7mMCxAAYMkzgQwAx5NM3nXzQUFdxA== --osd-uuid ca954ccf-6dcf-4629-a0b6-53289c25af93 2026-02-21T11:34:09.316 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:09.314+0000 7facac40b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:09.317 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:09.315+0000 7facac40b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:09.318 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:09.316+0000 7facac40b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:09.318 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:09.316+0000 7facac40b780 -1 bdev(0x2ec8c00 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:34:09.318 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:09.317+0000 7facac40b780 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2026-02-21T11:34:09.694 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/0/keyring 2026-02-21T11:34:09.695 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:34:09.695 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:34:09.696 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:34:09.696 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:34:09.796 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:34:09.797 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:34:09.797 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:34:09.797 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:34:09.797 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:34:09.797 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:34:09.809 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:09.808+0000 7f1dc1e39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:09.811 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:09.809+0000 7f1dc1e39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:09.812 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:09.810+0000 7f1dc1e39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:09.898 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:34:09.898 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:34:09.898 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:34:09.898 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:34:09.898 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:34:09.898 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:34:09.898 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:34:09.898 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:34:09.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:34:09.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:34:09.994 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:34:10.016 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:10.015+0000 7f1dc1e39780 -1 Falling back to public interface 2026-02-21T11:34:10.322 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:10.320+0000 7f1dc1e39780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:34:10.995 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:34:10.995 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:34:10.995 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:34:10.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:34:10.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:34:10.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:34:11.090 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/338313194,v1:127.0.0.1:6801/338313194] [v2:127.0.0.1:6802/338313194,v1:127.0.0.1:6803/338313194] exists,up ca954ccf-6dcf-4629-a0b6-53289c25af93 2026-02-21T11:34:11.093 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:34:11.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:34:11.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:34:11.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2026-02-21T11:34:11.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:570: run_mds: local dir=td/ok-to-stop 2026-02-21T11:34:11.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:571: run_mds: shift 2026-02-21T11:34:11.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:572: run_mds: local id=a 2026-02-21T11:34:11.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:573: run_mds: shift 2026-02-21T11:34:11.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:574: run_mds: local data=td/ok-to-stop/a 2026-02-21T11:34:11.095 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: get_asok_path 2026-02-21T11:34:11.095 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:11.095 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:11.095 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:11.095 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:11.095 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:11.095 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:11.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2026-02-21T11:34:11.105 INFO:tasks.workunit.client.0.trial189.stdout:starting mds.a at 2026-02-21T11:34:11.107 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2026-02-21T11:34:11.242 INFO:tasks.workunit.client.0.trial189.stderr:pool 'meta' already exists 2026-02-21T11:34:11.249 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2026-02-21T11:34:11.344 INFO:tasks.workunit.client.0.trial189.stderr:pool 'data' already exists 2026-02-21T11:34:11.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2026-02-21T11:34:11.730 INFO:tasks.workunit.client.0.trial189.stderr:filesystem 'myfs' already exists 2026-02-21T11:34:11.738 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2026-02-21T11:34:16.739 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2026-02-21T11:34:16.939 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4115 and/or standby gids 2026-02-21T11:34:16.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2026-02-21T11:34:17.140 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4115 and/or standby gids 2026-02-21T11:34:17.144 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2026-02-21T11:34:17.450 INFO:tasks.workunit.client.0.trial189.stderr:should be safe to stop [dne] 2026-02-21T11:34:17.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2026-02-21T11:34:17.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:570: run_mds: local dir=td/ok-to-stop 2026-02-21T11:34:17.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:571: run_mds: shift 2026-02-21T11:34:17.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:572: run_mds: local id=b 2026-02-21T11:34:17.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:573: run_mds: shift 2026-02-21T11:34:17.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:574: run_mds: local data=td/ok-to-stop/b 2026-02-21T11:34:17.458 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: get_asok_path 2026-02-21T11:34:17.458 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:17.458 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:17.458 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:17.458 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:17.458 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:17.459 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:17.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2026-02-21T11:34:17.472 INFO:tasks.workunit.client.0.trial189.stdout:starting mds.b at 2026-02-21T11:34:17.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2026-02-21T11:34:22.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2026-02-21T11:34:22.782 INFO:tasks.workunit.client.0.trial189.stderr:should be safe to stop [a] 2026-02-21T11:34:22.789 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2026-02-21T11:34:23.099 INFO:tasks.workunit.client.0.trial189.stderr:should be safe to stop [b] 2026-02-21T11:34:23.106 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2026-02-21T11:34:23.310 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4115 and/or standby gids 4135 2026-02-21T11:34:23.313 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2026-02-21T11:34:23.622 INFO:tasks.workunit.client.0.trial189.stderr:should be safe to stop [a,dne1,dne2] 2026-02-21T11:34:23.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2026-02-21T11:34:23.936 INFO:tasks.workunit.client.0.trial189.stderr:should be safe to stop [b,dne] 2026-02-21T11:34:23.943 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2026-02-21T11:34:24.148 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4115 and/or standby gids 4135 2026-02-21T11:34:24.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2026-02-21T11:34:24.461 INFO:tasks.workunit.client.0.trial189.stderr:should be safe to stop [dne1,dne2] 2026-02-21T11:34:24.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2026-02-21T11:34:24.469 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:24.469 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:24.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:24.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:24.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:24.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:24.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-21T11:34:24.575 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:24.576 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:24.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:24.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:24.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:24.684 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:24.685 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-21T11:34:24.685 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T11:34:24.685 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:34:24.685 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T11:34:24.685 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:24.685 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:24.685 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:24.685 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:24.685 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:24.691 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:24.691 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:34:24.692 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:34:24.692 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:34:24.693 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:34:24.693 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:34:24.693 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:34:24.694 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:24.694 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:34:24.694 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:34:24.694 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:24.695 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:34:24.696 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:34:24.696 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T11:34:24.700 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:34:24.700 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:24.701 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:24.701 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.58069 2026-02-21T11:34:24.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:34:24.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:34:24.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-21T11:34:24.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-21T11:34:24.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-21T11:34:24.703 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-21T11:34:24.703 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T11:34:24.703 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:34:24.703 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T11:34:24.703 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:24.703 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:24.703 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:24.703 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:24.703 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:24.704 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:24.704 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:34:24.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:34:24.705 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:34:24.706 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:34:24.706 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:34:24.706 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:34:24.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:24.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:34:24.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:34:24.707 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:24.708 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:34:24.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:34:24.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T11:34:24.710 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:34:24.710 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:24.710 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:24.710 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.58069 2026-02-21T11:34:24.711 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:34:24.711 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:34:24.711 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-21T11:34:24.712 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:34:24.712 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:24.712 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:24.712 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.58069 2026-02-21T11:34:24.713 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:34:24.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:34:24.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:34:24.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-21T11:34:24.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2026-02-21T11:34:24.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2026-02-21T11:34:24.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T11:34:24.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-21T11:34:24.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:34:24.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:34:24.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:34:24.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:34:24.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-21T11:34:24.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-21T11:34:24.732 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:34:24.732 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:24.732 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:24.732 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:24.732 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:24.732 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:24.732 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:24.732 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-21T11:34:24.756 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:34:24.758 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:34:24.759 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:24.759 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:34:24.759 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:34:24.759 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:34:24.759 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:34:24.759 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:34:24.759 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:34:24.759 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:24.759 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:24.760 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:24.760 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.a.asok 2026-02-21T11:34:24.760 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:24.760 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.a.asok config get fsid 2026-02-21T11:34:24.802 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:34:24.802 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:24.802 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:34:24.802 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:34:24.802 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:34:24.803 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:34:24.803 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:34:24.803 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:34:24.803 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:24.803 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:24.803 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:24.803 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.a.asok 2026-02-21T11:34:24.803 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:24.803 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.a.asok config get mon_host 2026-02-21T11:34:24.845 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2026-02-21T11:34:24.845 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-stop 2026-02-21T11:34:24.845 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:34:24.845 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:34:24.845 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:34:24.845 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-stop/x 2026-02-21T11:34:24.845 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:34:24.942 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:34:24.942 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:24.942 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:24.942 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:24.942 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:24.942 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:24.942 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:24.943 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:34:24.943 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:34:24.958 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2026-02-21T11:34:24.958 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-21T11:34:24.958 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:34:24.958 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:34:24.958 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:34:24.958 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/0 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:34:24.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/0 2026-02-21T11:34:24.960 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:34:24.960 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a305892f-cbc1-45dc-ad57-ed5b81a8de58 2026-02-21T11:34:24.960 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a305892f-cbc1-45dc-ad57-ed5b81a8de58' 2026-02-21T11:34:24.960 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 a305892f-cbc1-45dc-ad57-ed5b81a8de58 2026-02-21T11:34:24.960 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:34:24.969 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBAmJlpXT+6ORAAZRa9vfkEu8dP/f8ksd+UKA== 2026-02-21T11:34:24.969 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBAmJlpXT+6ORAAZRa9vfkEu8dP/f8ksd+UKA=="}' 2026-02-21T11:34:24.970 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a305892f-cbc1-45dc-ad57-ed5b81a8de58 -i td/ok-to-stop/0/new.json 2026-02-21T11:34:25.070 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:34:25.077 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/0/new.json 2026-02-21T11:34:25.078 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAmJlpXT+6ORAAZRa9vfkEu8dP/f8ksd+UKA== --osd-uuid a305892f-cbc1-45dc-ad57-ed5b81a8de58 2026-02-21T11:34:25.093 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:25.091+0000 7ff46a049780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:25.094 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:25.092+0000 7ff46a049780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:25.095 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:25.093+0000 7ff46a049780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:25.095 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:25.094+0000 7ff46a049780 -1 bdev(0x30d6c00 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:34:25.095 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:25.094+0000 7ff46a049780 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2026-02-21T11:34:25.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/0/keyring 2026-02-21T11:34:25.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:34:25.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:34:25.497 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:34:25.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:34:25.596 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:34:25.597 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:34:25.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:34:25.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:34:25.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:34:25.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:34:25.610 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:25.608+0000 7f392e646780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:25.612 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:25.610+0000 7f392e646780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:25.613 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:25.611+0000 7f392e646780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:25.693 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:34:25.693 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:34:25.693 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:34:25.694 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:34:25.694 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:34:25.694 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:34:25.694 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:34:25.694 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:34:25.694 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:34:25.694 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:34:25.789 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:34:25.838 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:25.836+0000 7f392e646780 -1 Falling back to public interface 2026-02-21T11:34:26.149 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:26.147+0000 7f392e646780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:34:26.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:34:26.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:34:26.790 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:34:26.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:34:26.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:34:26.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:34:26.885 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3601091987,v1:127.0.0.1:6801/3601091987] [v2:127.0.0.1:6802/3601091987,v1:127.0.0.1:6803/3601091987] exists,up a305892f-cbc1-45dc-ad57-ed5b81a8de58 2026-02-21T11:34:26.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:34:26.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:34:26.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:34:26.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2026-02-21T11:34:26.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-21T11:34:26.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:34:26.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T11:34:26.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:34:26.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/1 2026-02-21T11:34:26.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T11:34:26.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:34:26.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:34:26.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/1 2026-02-21T11:34:26.889 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:34:26.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=254da5ae-6070-4a02-b572-5daf02be2112 2026-02-21T11:34:26.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 254da5ae-6070-4a02-b572-5daf02be2112' 2026-02-21T11:34:26.889 INFO:tasks.workunit.client.0.trial189.stdout:add osd1 254da5ae-6070-4a02-b572-5daf02be2112 2026-02-21T11:34:26.889 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:34:26.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBCmJlpMUmGNRAAw6K7R0i1BAIx/UZzYy3rKA== 2026-02-21T11:34:26.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBCmJlpMUmGNRAAw6K7R0i1BAIx/UZzYy3rKA=="}' 2026-02-21T11:34:26.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 254da5ae-6070-4a02-b572-5daf02be2112 -i td/ok-to-stop/1/new.json 2026-02-21T11:34:27.000 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:34:27.006 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/1/new.json 2026-02-21T11:34:27.007 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCmJlpMUmGNRAAw6K7R0i1BAIx/UZzYy3rKA== --osd-uuid 254da5ae-6070-4a02-b572-5daf02be2112 2026-02-21T11:34:27.022 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:27.020+0000 7f9f08453780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:27.023 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:27.022+0000 7f9f08453780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:27.024 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:27.023+0000 7f9f08453780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:27.025 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:27.023+0000 7f9f08453780 -1 bdev(0x3fbec00 td/ok-to-stop/1/block) open stat got: (1) Operation not permitted 2026-02-21T11:34:27.025 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:27.023+0000 7f9f08453780 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2026-02-21T11:34:27.348 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/1/keyring 2026-02-21T11:34:27.348 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:34:27.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T11:34:27.349 INFO:tasks.workunit.client.0.trial189.stdout:adding osd1 key to auth repository 2026-02-21T11:34:27.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:34:27.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T11:34:27.662 INFO:tasks.workunit.client.0.trial189.stdout:start osd.1 2026-02-21T11:34:27.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:34:27.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:34:27.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:34:27.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:34:27.675 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:27.673+0000 7fa037e35780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:27.676 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:27.674+0000 7fa037e35780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:27.677 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:27.675+0000 7fa037e35780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:27.806 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:27.804+0000 7fa037e35780 -1 Falling back to public interface 2026-02-21T11:34:27.885 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T11:34:27.886 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:34:27.886 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T11:34:27.886 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:34:27.886 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:34:27.886 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:34:27.886 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:34:27.886 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:34:27.886 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:34:27.886 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:34:28.105 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:28.103+0000 7fa037e35780 -1 osd.1 0 log_to_monitors true 2026-02-21T11:34:28.116 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:34:29.118 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:34:29.118 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:34:29.119 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:34:29.120 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:34:29.120 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:34:29.120 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:34:29.344 INFO:tasks.workunit.client.0.trial189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/569324782,v1:127.0.0.1:6811/569324782] [v2:127.0.0.1:6812/569324782,v1:127.0.0.1:6813/569324782] exists,up 254da5ae-6070-4a02-b572-5daf02be2112 2026-02-21T11:34:29.348 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:34:29.348 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:34:29.348 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:34:29.348 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2026-02-21T11:34:29.348 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-21T11:34:29.348 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:34:29.348 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T11:34:29.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:34:29.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/2 2026-02-21T11:34:29.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T11:34:29.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:34:29.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:34:29.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:34:29.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2026-02-21T11:34:29.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2026-02-21T11:34:29.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:34:29.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:34:29.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-21T11:34:29.349 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:34:29.350 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:29.350 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:29.350 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:29.350 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:29.350 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:29.350 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:29.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:29.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:34:29.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:34:29.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:34:29.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-21T11:34:29.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-21T11:34:29.351 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:34:29.351 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:34:29.351 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:34:29.351 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:34:29.351 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:34:29.351 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:34:29.351 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/2 2026-02-21T11:34:29.351 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:34:29.351 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=622c316b-8660-4534-9006-1d8101b7044b 2026-02-21T11:34:29.351 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 622c316b-8660-4534-9006-1d8101b7044b' 2026-02-21T11:34:29.352 INFO:tasks.workunit.client.0.trial189.stdout:add osd2 622c316b-8660-4534-9006-1d8101b7044b 2026-02-21T11:34:29.352 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:34:29.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBFmJlptIsyFRAA44nLrZGno2bhvH15dyF5uA== 2026-02-21T11:34:29.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBFmJlptIsyFRAA44nLrZGno2bhvH15dyF5uA=="}' 2026-02-21T11:34:29.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 622c316b-8660-4534-9006-1d8101b7044b -i td/ok-to-stop/2/new.json 2026-02-21T11:34:29.585 INFO:tasks.workunit.client.0.trial189.stdout:2 2026-02-21T11:34:29.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/2/new.json 2026-02-21T11:34:29.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBFmJlptIsyFRAA44nLrZGno2bhvH15dyF5uA== --osd-uuid 622c316b-8660-4534-9006-1d8101b7044b 2026-02-21T11:34:29.606 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:29.604+0000 7f3d80e2d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:29.608 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:29.606+0000 7f3d80e2d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:29.608 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:29.607+0000 7f3d80e2d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:29.609 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:29.607+0000 7f3d80e2d780 -1 bdev(0x407ac00 td/ok-to-stop/2/block) open stat got: (1) Operation not permitted 2026-02-21T11:34:29.609 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:29.607+0000 7f3d80e2d780 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2026-02-21T11:34:29.938 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/2/keyring 2026-02-21T11:34:29.939 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:34:29.939 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T11:34:29.939 INFO:tasks.workunit.client.0.trial189.stdout:adding osd2 key to auth repository 2026-02-21T11:34:29.939 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:34:30.247 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T11:34:30.247 INFO:tasks.workunit.client.0.trial189.stdout:start osd.2 2026-02-21T11:34:30.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:34:30.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:34:30.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:34:30.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:34:30.261 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:30.260+0000 7f79b8239780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:30.263 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:30.261+0000 7f79b8239780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:30.263 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:30.262+0000 7f79b8239780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:30.404 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:30.402+0000 7f79b8239780 -1 Falling back to public interface 2026-02-21T11:34:30.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T11:34:30.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:34:30.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T11:34:30.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:34:30.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:34:30.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:34:30.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:34:30.475 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:34:30.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:34:30.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T11:34:30.699 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:34:30.781 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:30.779+0000 7f79b8239780 -1 osd.2 0 log_to_monitors true 2026-02-21T11:34:31.700 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:34:31.700 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:34:31.701 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:34:31.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:34:31.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:34:31.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T11:34:31.927 INFO:tasks.workunit.client.0.trial189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3752896481,v1:127.0.0.1:6819/3752896481] [v2:127.0.0.1:6820/3752896481,v1:127.0.0.1:6821/3752896481] exists,up 622c316b-8660-4534-9006-1d8101b7044b 2026-02-21T11:34:31.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/3 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:31.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/3 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b47b1cb5-9293-4bbe-8092-878a96936a61 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 b47b1cb5-9293-4bbe-8092-878a96936a61' 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stdout:add osd3 b47b1cb5-9293-4bbe-8092-878a96936a61 2026-02-21T11:34:31.932 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:34:31.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBHmJlpE7D7NxAAu/dMpaUDRxb2HuMlI2iL5Q== 2026-02-21T11:34:31.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBHmJlpE7D7NxAAu/dMpaUDRxb2HuMlI2iL5Q=="}' 2026-02-21T11:34:31.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b47b1cb5-9293-4bbe-8092-878a96936a61 -i td/ok-to-stop/3/new.json 2026-02-21T11:34:32.168 INFO:tasks.workunit.client.0.trial189.stdout:3 2026-02-21T11:34:32.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/3/new.json 2026-02-21T11:34:32.177 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBHmJlpE7D7NxAAu/dMpaUDRxb2HuMlI2iL5Q== --osd-uuid b47b1cb5-9293-4bbe-8092-878a96936a61 2026-02-21T11:34:32.191 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:32.190+0000 7fa45c635780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:32.193 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:32.191+0000 7fa45c635780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:32.193 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:32.192+0000 7fa45c635780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:32.194 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:32.192+0000 7fa45c635780 -1 bdev(0x410ec00 td/ok-to-stop/3/block) open stat got: (1) Operation not permitted 2026-02-21T11:34:32.194 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:32.192+0000 7fa45c635780 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2026-02-21T11:34:32.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/3/keyring 2026-02-21T11:34:32.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:34:32.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-21T11:34:32.519 INFO:tasks.workunit.client.0.trial189.stdout:adding osd3 key to auth repository 2026-02-21T11:34:32.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:34:32.831 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-21T11:34:32.832 INFO:tasks.workunit.client.0.trial189.stdout:start osd.3 2026-02-21T11:34:32.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:34:32.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:34:32.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:34:32.833 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:34:32.845 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:32.843+0000 7efd6fa33780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:32.847 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:32.845+0000 7efd6fa33780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:32.847 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:32.846+0000 7efd6fa33780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:34:32.994 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:32.992+0000 7efd6fa33780 -1 Falling back to public interface 2026-02-21T11:34:33.057 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-21T11:34:33.057 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:34:33.057 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-21T11:34:33.057 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:34:33.057 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:34:33.057 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:34:33.058 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:34:33.058 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:34:33.058 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:34:33.059 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-21T11:34:33.284 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:34:33.285 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:34:33.283+0000 7efd6fa33780 -1 osd.3 0 log_to_monitors true 2026-02-21T11:34:34.285 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:34:34.286 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:34:34.286 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:34:34.287 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:34:34.287 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:34:34.287 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-21T11:34:34.510 INFO:tasks.workunit.client.0.trial189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/728150661,v1:127.0.0.1:6827/728150661] [v2:127.0.0.1:6828/728150661,v1:127.0.0.1:6829/728150661] exists,up b47b1cb5-9293-4bbe-8092-878a96936a61 2026-02-21T11:34:34.511 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:34:34.511 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:34:34.511 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:34:34.511 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2026-02-21T11:34:34.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2026-02-21T11:34:35.142 INFO:tasks.workunit.client.0.trial189.stderr:pool 'ec' already exists 2026-02-21T11:34:35.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2026-02-21T11:34:35.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:34:35.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:34:35.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:34:35.152 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:34:35.152 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:34:35.153 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:34:35.153 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:34:35.153 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:34:35.153 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:34:35.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:34:35.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:34:35.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:34:35.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:34:35.241 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:34:35.241 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:34:35.466 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:34:35.466 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:34:35.466 INFO:tasks.workunit.client.0.trial189.stderr:2 2026-02-21T11:34:35.466 INFO:tasks.workunit.client.0.trial189.stderr:3' 2026-02-21T11:34:35.466 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:34:35.467 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:35.467 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:34:35.531 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836484 2026-02-21T11:34:35.531 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836484 2026-02-21T11:34:35.531 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836484' 2026-02-21T11:34:35.531 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:35.531 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:34:35.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=42949672964 2026-02-21T11:34:35.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 42949672964 2026-02-21T11:34:35.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836484 1-42949672964' 2026-02-21T11:34:35.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:35.594 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-21T11:34:35.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=64424509443 2026-02-21T11:34:35.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 64424509443 2026-02-21T11:34:35.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836484 1-42949672964 2-64424509443' 2026-02-21T11:34:35.657 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:35.657 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-21T11:34:35.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=81604378627 2026-02-21T11:34:35.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 81604378627 2026-02-21T11:34:35.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836484 1-42949672964 2-64424509443 3-81604378627' 2026-02-21T11:34:35.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:35.719 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836484 2026-02-21T11:34:35.719 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:35.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:34:35.721 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836484 2026-02-21T11:34:35.721 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:35.722 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836484 2026-02-21T11:34:35.722 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-21T11:34:35.722 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836484 2026-02-21T11:34:35.722 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:34:35.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-21T11:34:35.948 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:34:36.948 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:34:36.948 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:34:37.174 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-21T11:34:37.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:37.175 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-42949672964 2026-02-21T11:34:37.175 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:37.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:34:37.176 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-42949672964 2026-02-21T11:34:37.176 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:37.178 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=42949672964 2026-02-21T11:34:37.178 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2026-02-21T11:34:37.178 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 42949672964 2026-02-21T11:34:37.178 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:34:37.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 42949672964 -lt 42949672964 2026-02-21T11:34:37.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:37.406 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 2-64424509443 2026-02-21T11:34:37.406 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:37.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=2 2026-02-21T11:34:37.408 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 2-64424509443 2026-02-21T11:34:37.408 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:37.409 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=64424509443 2026-02-21T11:34:37.409 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.2 seq 64424509443' 2026-02-21T11:34:37.409 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.2 seq 64424509443 2026-02-21T11:34:37.410 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T11:34:37.635 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 64424509443 -lt 64424509443 2026-02-21T11:34:37.635 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:37.636 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 3-81604378627 2026-02-21T11:34:37.636 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:37.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=3 2026-02-21T11:34:37.637 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 3-81604378627 2026-02-21T11:34:37.637 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:37.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=81604378627 2026-02-21T11:34:37.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.3 seq 81604378627' 2026-02-21T11:34:37.639 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.3 seq 81604378627 2026-02-21T11:34:37.639 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-21T11:34:37.868 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 81604378627 -lt 81604378627 2026-02-21T11:34:37.869 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:34:37.869 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:34:37.869 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:34:38.184 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-21T11:34:38.184 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:34:38.184 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:34:38.184 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:34:38.185 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:34:38.185 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:34:38.185 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:34:38.185 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:34:38.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-21T11:34:38.407 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:34:38.408 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:34:38.408 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:34:38.716 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-21T11:34:38.716 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:34:38.717 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:34:38.717 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2026-02-21T11:34:39.027 INFO:tasks.workunit.client.0.trial189.stderr:set pool 1 min_size to 3 2026-02-21T11:34:39.036 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2026-02-21T11:34:39.037 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:34:39.037 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:34:39.037 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:34:39.037 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:34:39.037 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:34:39.038 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:34:39.038 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:34:39.038 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:34:39.038 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:34:39.120 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:34:39.120 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:34:39.120 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:34:39.120 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:34:39.120 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:34:39.120 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:34:39.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:34:39.347 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:34:39.347 INFO:tasks.workunit.client.0.trial189.stderr:2 2026-02-21T11:34:39.347 INFO:tasks.workunit.client.0.trial189.stderr:3' 2026-02-21T11:34:39.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:34:39.348 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:39.348 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:34:39.411 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836487 2026-02-21T11:34:39.411 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836487 2026-02-21T11:34:39.411 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836487' 2026-02-21T11:34:39.411 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:39.411 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:34:39.473 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=42949672967 2026-02-21T11:34:39.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 42949672967 2026-02-21T11:34:39.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836487 1-42949672967' 2026-02-21T11:34:39.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:39.474 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-21T11:34:39.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=64424509446 2026-02-21T11:34:39.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 64424509446 2026-02-21T11:34:39.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836487 1-42949672967 2-64424509446' 2026-02-21T11:34:39.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:39.537 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-21T11:34:39.601 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=81604378630 2026-02-21T11:34:39.601 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 81604378630 2026-02-21T11:34:39.601 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836487 1-42949672967 2-64424509446 3-81604378630' 2026-02-21T11:34:39.601 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:39.601 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836487 2026-02-21T11:34:39.601 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:39.602 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:34:39.602 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836487 2026-02-21T11:34:39.602 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:39.603 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836487 2026-02-21T11:34:39.603 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2026-02-21T11:34:39.603 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836487 2026-02-21T11:34:39.604 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:34:39.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836485 -lt 21474836487 2026-02-21T11:34:39.829 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:34:40.829 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:34:40.830 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:34:41.057 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836487 -lt 21474836487 2026-02-21T11:34:41.057 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:41.058 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-42949672967 2026-02-21T11:34:41.058 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:41.058 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:34:41.059 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-42949672967 2026-02-21T11:34:41.059 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:41.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=42949672967 2026-02-21T11:34:41.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-21T11:34:41.060 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 42949672967 2026-02-21T11:34:41.061 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:34:41.288 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 42949672967 -lt 42949672967 2026-02-21T11:34:41.288 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:41.288 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 2-64424509446 2026-02-21T11:34:41.288 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:41.289 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=2 2026-02-21T11:34:41.289 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 2-64424509446 2026-02-21T11:34:41.289 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:41.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=64424509446 2026-02-21T11:34:41.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2026-02-21T11:34:41.291 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.2 seq 64424509446 2026-02-21T11:34:41.291 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T11:34:41.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 64424509446 -lt 64424509446 2026-02-21T11:34:41.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:41.518 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 3-81604378630 2026-02-21T11:34:41.518 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:41.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=3 2026-02-21T11:34:41.519 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 3-81604378630 2026-02-21T11:34:41.519 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:41.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=81604378630 2026-02-21T11:34:41.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.3 seq 81604378630' 2026-02-21T11:34:41.521 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.3 seq 81604378630 2026-02-21T11:34:41.521 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-21T11:34:41.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 81604378630 -lt 81604378630 2026-02-21T11:34:41.744 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:34:41.744 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:34:41.744 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:34:42.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-21T11:34:42.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:34:42.054 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:34:42.054 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:34:42.054 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:34:42.054 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:34:42.054 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:34:42.054 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:34:42.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-21T11:34:42.274 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:34:42.274 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:34:42.275 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:34:42.586 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-21T11:34:42.586 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:34:42.586 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:34:42.586 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2026-02-21T11:34:42.801 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":6,"num_not_ok_pgs":0,"ok_become_degraded":["1.7","1.8","1.9","1.a","1.d","1.1a"]} 2026-02-21T11:34:42.808 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2026-02-21T11:34:43.025 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":6,"num_not_ok_pgs":0,"ok_become_degraded":["1.4","1.5","1.c","1.14","1.18","1.19"]} 2026-02-21T11:34:43.033 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2026-02-21T11:34:43.248 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":15,"num_not_ok_pgs":0,"ok_become_degraded":["1.1","1.2","1.6","1.e","1.f","1.11","1.12","1.13","1.15","1.16","1.17","1.1b","1.1c","1.1d","1.1e"]} 2026-02-21T11:34:43.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2026-02-21T11:34:43.470 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":5,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.3","1.b","1.10","1.1f"]} 2026-02-21T11:34:43.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2026-02-21T11:34:43.631 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":12,"bad_become_inactive":["1.4","1.5","1.7","1.8","1.9","1.a","1.c","1.d","1.14","1.18","1.19","1.1a"]} 2026-02-21T11:34:43.631 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2026-02-21T11:34:43.634 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2026-02-21T11:34:43.792 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":20,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.6","1.b","1.e","1.f","1.10","1.11","1.12","1.13","1.15","1.16","1.17","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T11:34:43.792 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: unsafe to stop osd(s) at this time (20 PGs are or would become offline) 2026-02-21T11:34:43.796 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2026-02-21T11:34:43.796 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2026-02-21T11:34:44.024 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":6,"num_not_ok_pgs":0,"ok_become_degraded":["1.7","1.8","1.9","1.a","1.d","1.1a"]} 2026-02-21T11:34:44.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2026-02-21T11:34:44.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2026-02-21T11:34:44.245 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":6,"num_not_ok_pgs":0,"ok_become_degraded":["1.4","1.5","1.c","1.14","1.18","1.19"]} 2026-02-21T11:34:44.245 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2026-02-21T11:34:44.553 INFO:tasks.workunit.client.0.trial189.stderr:set pool 1 min_size to 2 2026-02-21T11:34:44.565 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2026-02-21T11:34:44.566 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:34:44.566 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:34:44.566 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:34:44.566 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:34:44.567 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:34:44.567 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:34:44.567 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:34:44.567 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:34:44.567 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:34:44.649 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:34:44.649 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:34:44.649 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:34:44.649 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:34:44.649 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:34:44.649 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:34:44.873 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:34:44.873 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:34:44.873 INFO:tasks.workunit.client.0.trial189.stderr:2 2026-02-21T11:34:44.873 INFO:tasks.workunit.client.0.trial189.stderr:3' 2026-02-21T11:34:44.873 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:34:44.873 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:44.873 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:34:44.937 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836490 2026-02-21T11:34:44.937 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836490 2026-02-21T11:34:44.937 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836490' 2026-02-21T11:34:44.937 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:44.937 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:34:44.999 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=42949672970 2026-02-21T11:34:45.000 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 42949672970 2026-02-21T11:34:45.000 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836490 1-42949672970' 2026-02-21T11:34:45.000 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:45.000 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-21T11:34:45.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=64424509449 2026-02-21T11:34:45.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 64424509449 2026-02-21T11:34:45.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836490 1-42949672970 2-64424509449' 2026-02-21T11:34:45.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:45.063 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-21T11:34:45.126 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=81604378633 2026-02-21T11:34:45.126 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 81604378633 2026-02-21T11:34:45.126 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836490 1-42949672970 2-64424509449 3-81604378633' 2026-02-21T11:34:45.127 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:45.127 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836490 2026-02-21T11:34:45.127 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:45.127 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:34:45.128 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836490 2026-02-21T11:34:45.128 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:45.129 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836490 2026-02-21T11:34:45.129 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-21T11:34:45.129 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836490 2026-02-21T11:34:45.130 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:34:45.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836490 -lt 21474836490 2026-02-21T11:34:45.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:45.356 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-42949672970 2026-02-21T11:34:45.356 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:45.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:34:45.357 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-42949672970 2026-02-21T11:34:45.357 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:45.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=42949672970 2026-02-21T11:34:45.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2026-02-21T11:34:45.360 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 42949672970 2026-02-21T11:34:45.360 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:34:45.582 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 42949672968 -lt 42949672970 2026-02-21T11:34:45.582 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:34:46.583 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:34:46.584 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:34:46.807 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 42949672968 -lt 42949672970 2026-02-21T11:34:46.807 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:34:47.808 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T11:34:47.808 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:34:48.035 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 42949672970 -lt 42949672970 2026-02-21T11:34:48.036 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:48.036 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 2-64424509449 2026-02-21T11:34:48.036 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:48.037 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=2 2026-02-21T11:34:48.037 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 2-64424509449 2026-02-21T11:34:48.037 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:48.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=64424509449 2026-02-21T11:34:48.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2026-02-21T11:34:48.039 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.2 seq 64424509449 2026-02-21T11:34:48.039 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T11:34:48.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 64424509449 -lt 64424509449 2026-02-21T11:34:48.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:48.268 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 3-81604378633 2026-02-21T11:34:48.268 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:48.269 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=3 2026-02-21T11:34:48.269 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 3-81604378633 2026-02-21T11:34:48.269 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:48.270 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=81604378633 2026-02-21T11:34:48.271 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.3 seq 81604378633' 2026-02-21T11:34:48.271 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.3 seq 81604378633 2026-02-21T11:34:48.271 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-21T11:34:48.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 81604378633 -lt 81604378633 2026-02-21T11:34:48.500 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:34:48.500 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:34:48.500 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:34:48.813 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-21T11:34:48.814 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:34:48.814 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:34:48.814 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:34:48.814 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:34:48.814 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:34:48.814 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:34:48.814 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:34:49.039 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-21T11:34:49.039 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:34:49.039 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:34:49.039 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:34:49.353 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-21T11:34:49.353 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:34:49.353 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:34:49.353 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2026-02-21T11:34:49.568 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":12,"num_not_ok_pgs":0,"ok_become_degraded":["1.4","1.5","1.7","1.8","1.9","1.a","1.c","1.d","1.14","1.18","1.19","1.1a"]} 2026-02-21T11:34:49.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2026-02-21T11:34:49.790 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":20,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.6","1.b","1.e","1.f","1.10","1.11","1.12","1.13","1.15","1.16","1.17","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T11:34:49.797 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2026-02-21T11:34:49.956 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":27,"bad_become_inactive":["1.1","1.2","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.c","1.d","1.e","1.f","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e"]} 2026-02-21T11:34:49.957 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: unsafe to stop osd(s) at this time (27 PGs are or would become offline) 2026-02-21T11:34:49.958 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2026-02-21T11:34:50.115 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":26,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.b","1.c","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T11:34:50.115 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: unsafe to stop osd(s) at this time (26 PGs are or would become offline) 2026-02-21T11:34:50.119 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2026-02-21T11:34:50.119 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2026-02-21T11:34:50.344 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":12,"num_not_ok_pgs":0,"ok_become_degraded":["1.4","1.5","1.7","1.8","1.9","1.a","1.c","1.d","1.14","1.18","1.19","1.1a"]} 2026-02-21T11:34:50.345 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2026-02-21T11:34:50.345 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2026-02-21T11:34:50.568 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":12,"num_not_ok_pgs":0,"ok_become_degraded":["1.4","1.5","1.7","1.8","1.9","1.a","1.c","1.d","1.14","1.18","1.19","1.1a"]} 2026-02-21T11:34:50.569 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2026-02-21T11:34:50.569 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2026-02-21T11:34:50.795 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":21,"num_not_ok_pgs":0,"ok_become_degraded":["1.1","1.2","1.6","1.7","1.8","1.9","1.a","1.d","1.e","1.f","1.11","1.12","1.13","1.15","1.16","1.17","1.1a","1.1b","1.1c","1.1d","1.1e"]} 2026-02-21T11:34:50.796 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2026-02-21T11:34:50.796 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2026-02-21T11:34:51.023 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":21,"num_not_ok_pgs":0,"ok_become_degraded":["1.1","1.2","1.6","1.7","1.8","1.9","1.a","1.d","1.e","1.f","1.11","1.12","1.13","1.15","1.16","1.17","1.1a","1.1b","1.1c","1.1d","1.1e"]} 2026-02-21T11:34:51.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2026-02-21T11:34:51.024 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:51.024 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:51.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:51.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:51.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:51.331 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:51.331 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2026-02-21T11:34:51.560 INFO:tasks.workunit.client.0.trial189.stderr:osd.0 is already down. 2026-02-21T11:34:51.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2026-02-21T11:34:51.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1760: wait_for_peered: local cmd= 2026-02-21T11:34:51.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1761: wait_for_peered: local num_peered=-1 2026-02-21T11:34:51.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_peered: local cur_peered 2026-02-21T11:34:51.568 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_peered: get_timeout_delays 90 .1 2026-02-21T11:34:51.569 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:34:51.569 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:34:51.569 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:34:51.569 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:34:51.569 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:34:51.653 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:34:51.653 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1763: wait_for_peered: local -a delays 2026-02-21T11:34:51.653 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1764: wait_for_peered: local -i loop=0 2026-02-21T11:34:51.653 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1766: wait_for_peered: flush_pg_stats 2026-02-21T11:34:51.654 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:34:51.654 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:34:51.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:34:51.877 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:34:51.877 INFO:tasks.workunit.client.0.trial189.stderr:2 2026-02-21T11:34:51.877 INFO:tasks.workunit.client.0.trial189.stderr:3' 2026-02-21T11:34:51.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:34:51.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:51.877 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:34:51.922 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq= 2026-02-21T11:34:51.922 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: true 2026-02-21T11:34:51.922 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z '' 2026-02-21T11:34:51.922 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: flush_pg_stats: continue 2026-02-21T11:34:51.922 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:51.922 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:34:51.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=42949672973 2026-02-21T11:34:51.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 42949672973 2026-02-21T11:34:51.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 1-42949672973' 2026-02-21T11:34:51.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:51.988 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-21T11:34:52.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=64424509453 2026-02-21T11:34:52.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 64424509453 2026-02-21T11:34:52.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 1-42949672973 2-64424509453' 2026-02-21T11:34:52.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:34:52.050 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-21T11:34:52.114 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=81604378636 2026-02-21T11:34:52.115 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 81604378636 2026-02-21T11:34:52.115 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 1-42949672973 2-64424509453 3-81604378636' 2026-02-21T11:34:52.115 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:52.115 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-42949672973 2026-02-21T11:34:52.115 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:52.116 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:34:52.116 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-42949672973 2026-02-21T11:34:52.116 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:52.117 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=42949672973 2026-02-21T11:34:52.117 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2026-02-21T11:34:52.117 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 42949672973 2026-02-21T11:34:52.117 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:34:52.345 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 42949672971 -lt 42949672973 2026-02-21T11:34:52.345 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:34:53.346 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:34:53.347 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:34:53.572 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 42949672973 -lt 42949672973 2026-02-21T11:34:53.573 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:53.573 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 2-64424509453 2026-02-21T11:34:53.574 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:53.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=2 2026-02-21T11:34:53.575 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 2-64424509453 2026-02-21T11:34:53.575 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:53.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=64424509453 2026-02-21T11:34:53.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2026-02-21T11:34:53.577 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.2 seq 64424509453 2026-02-21T11:34:53.577 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T11:34:53.799 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 64424509453 -lt 64424509453 2026-02-21T11:34:53.799 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:34:53.799 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 3-81604378636 2026-02-21T11:34:53.799 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:34:53.800 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=3 2026-02-21T11:34:53.801 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 3-81604378636 2026-02-21T11:34:53.801 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:34:53.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=81604378636 2026-02-21T11:34:53.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.3 seq 81604378636' 2026-02-21T11:34:53.802 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.3 seq 81604378636 2026-02-21T11:34:53.802 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-21T11:34:54.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 81604378636 -lt 81604378636 2026-02-21T11:34:54.028 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_peered: get_num_pgs 2026-02-21T11:34:54.028 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:34:54.028 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:34:54.341 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1767: wait_for_peered: test 32 == 0 2026-02-21T11:34:54.342 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1771: wait_for_peered: true 2026-02-21T11:34:54.342 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1775: wait_for_peered: get_num_active_or_peered 2026-02-21T11:34:54.342 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1385: get_num_active_or_peered: local expression 2026-02-21T11:34:54.342 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1386: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2026-02-21T11:34:54.342 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1387: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2026-02-21T11:34:54.342 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1388: get_num_active_or_peered: ceph --format json pg dump pgs 2026-02-21T11:34:54.342 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1389: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2026-02-21T11:34:54.567 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1775: wait_for_peered: cur_peered=32 2026-02-21T11:34:54.567 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1776: wait_for_peered: get_num_pgs 2026-02-21T11:34:54.567 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:34:54.567 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:34:54.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1776: wait_for_peered: test 32 = 32 2026-02-21T11:34:54.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1776: wait_for_peered: break 2026-02-21T11:34:54.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_peered: return 0 2026-02-21T11:34:54.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2026-02-21T11:34:55.106 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2026-02-21T11:34:55.113 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2026-02-21T11:34:55.328 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":6,"num_not_ok_pgs":0,"ok_become_degraded":["1.4","1.5","1.c","1.14","1.18","1.19"]} 2026-02-21T11:34:55.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2026-02-21T11:34:55.492 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":21,"bad_become_inactive":["1.1","1.2","1.4","1.5","1.6","1.c","1.e","1.f","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1b","1.1c","1.1d","1.1e"]} 2026-02-21T11:34:55.493 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: unsafe to stop osd(s) at this time (21 PGs are or would become offline) 2026-02-21T11:34:55.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2026-02-21T11:34:55.654 INFO:tasks.workunit.client.0.trial189.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":26,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.b","1.c","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-21T11:34:55.654 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: unsafe to stop osd(s) at this time (26 PGs are or would become offline) 2026-02-21T11:34:55.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-21T11:34:55.658 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:55.659 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:55.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:55.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:55.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:55.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:55.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-21T11:34:55.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T11:34:55.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:34:55.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T11:34:55.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:55.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:55.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:55.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:55.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:55.773 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:55.773 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:34:55.774 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:34:55.774 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:34:55.775 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:34:55.775 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:34:55.775 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:34:55.776 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:55.776 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:34:55.777 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:34:55.777 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:55.777 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:34:55.778 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:34:55.778 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T11:34:55.792 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:34:55.792 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:55.792 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:55.792 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.58069 2026-02-21T11:34:55.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:34:55.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:34:55.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-21T11:34:55.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-21T11:34:55.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-21T11:34:55.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-21T11:34:55.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T11:34:55.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:34:55.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T11:34:55.794 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:55.794 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:55.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:55.795 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:55.795 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:55.796 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:55.796 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:34:55.797 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:34:55.797 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:34:55.798 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:34:55.798 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:34:55.798 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:34:55.799 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:55.799 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:34:55.799 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:34:55.800 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:55.800 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:34:55.801 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:34:55.801 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T11:34:55.802 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:34:55.802 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:55.802 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:55.803 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.58069 2026-02-21T11:34:55.803 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:34:55.803 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:34:55.804 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-21T11:34:55.805 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:34:55.805 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:55.805 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:55.805 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.58069 2026-02-21T11:34:55.806 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:34:55.806 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:34:55.806 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:34:55.806 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-21T11:34:55.807 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2026-02-21T11:34:55.807 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2026-02-21T11:34:55.807 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-21T11:34:55.807 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-21T11:34:55.807 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:34:55.807 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:34:55.807 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:34:55.807 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:34:55.807 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-21T11:34:55.807 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-21T11:34:55.826 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:34:55.826 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:55.826 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:55.826 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:55.826 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:55.826 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:55.826 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:55.826 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-21T11:34:55.849 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:34:55.850 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:34:55.850 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:55.850 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:34:55.850 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:34:55.850 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:34:55.850 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:34:55.850 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:34:55.850 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:34:55.850 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:55.850 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:55.850 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:55.851 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.a.asok 2026-02-21T11:34:55.851 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:55.851 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.a.asok config get fsid 2026-02-21T11:34:55.893 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:34:55.893 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:55.893 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:34:55.893 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:34:55.894 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:34:55.894 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:34:55.894 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:34:55.894 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:34:55.894 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:55.894 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:55.894 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:55.894 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.a.asok 2026-02-21T11:34:55.894 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:55.894 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.a.asok config get mon_host 2026-02-21T11:34:55.937 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2026-02-21T11:34:56.028 INFO:tasks.workunit.client.0.trial189.stderr:invalid mon(s) specified: [dne] 2026-02-21T11:34:56.033 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2026-02-21T11:34:56.105 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a] 2026-02-21T11:34:56.108 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2026-02-21T11:34:56.178 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2026-02-21T11:34:56.180 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2026-02-21T11:34:56.255 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: removing mon.a would break quorum 2026-02-21T11:34:56.258 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2026-02-21T11:34:56.349 INFO:tasks.workunit.client.0.trial189.stderr:mon.dne does not exist 2026-02-21T11:34:56.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-21T11:34:56.354 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:56.354 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:56.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:56.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:56.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:56.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:56.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-21T11:34:56.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T11:34:56.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:34:56.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T11:34:56.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:56.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:56.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:56.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:56.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:56.467 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:56.467 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:34:56.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:34:56.468 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:34:56.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:34:56.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:34:56.469 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:34:56.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:56.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:34:56.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:34:56.470 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:56.471 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:34:56.472 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:34:56.472 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T11:34:56.474 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:34:56.474 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:56.474 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:56.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.58069 2026-02-21T11:34:56.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:34:56.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:34:56.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-21T11:34:56.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-21T11:34:56.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-21T11:34:56.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-21T11:34:56.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T11:34:56.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:34:56.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T11:34:56.477 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:56.477 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:56.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:56.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:56.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:56.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:56.478 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:34:56.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:34:56.478 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:34:56.479 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:34:56.479 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:34:56.480 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:34:56.480 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:56.480 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:34:56.481 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:34:56.481 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:56.482 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:34:56.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:34:56.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T11:34:56.484 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:34:56.484 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:56.484 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:56.484 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.58069 2026-02-21T11:34:56.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:34:56.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:34:56.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-21T11:34:56.486 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:34:56.486 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:56.486 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:56.487 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.58069 2026-02-21T11:34:56.487 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:34:56.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:34:56.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:34:56.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-21T11:34:56.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2026-02-21T11:34:56.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2026-02-21T11:34:56.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2026-02-21T11:34:56.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-21T11:34:56.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:34:56.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:34:56.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:34:56.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:34:56.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-21T11:34:56.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-21T11:34:56.506 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:34:56.506 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:56.506 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:56.506 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:56.506 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:56.506 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:56.506 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:56.506 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-21T11:34:56.529 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:34:56.530 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:34:56.530 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:56.530 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:34:56.530 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:34:56.530 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:34:56.531 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:34:56.531 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:34:56.531 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:34:56.531 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:56.531 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:56.531 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:56.531 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.a.asok 2026-02-21T11:34:56.531 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:56.531 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.a.asok config get fsid 2026-02-21T11:34:56.576 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:34:56.576 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:56.576 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:34:56.576 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:34:56.576 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:34:56.576 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:34:56.576 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:34:56.576 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:34:56.576 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:56.577 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:56.577 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:56.577 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.a.asok 2026-02-21T11:34:56.577 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:56.577 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.a.asok config get mon_host 2026-02-21T11:34:56.617 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-21T11:34:56.617 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:34:56.617 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:34:56.617 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-21T11:34:56.617 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:34:56.617 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-21T11:34:56.617 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-21T11:34:56.634 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:34:56.634 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:56.634 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:56.634 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:56.634 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:56.634 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:56.634 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:56.635 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-21T11:34:56.657 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:34:56.658 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-21T11:34:56.658 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:56.658 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T11:34:56.658 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:34:56.658 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:34:56.659 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T11:34:56.659 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T11:34:56.659 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T11:34:56.659 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:56.659 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:56.659 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:56.659 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.b.asok 2026-02-21T11:34:56.659 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:56.660 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.b.asok config get fsid 2026-02-21T11:34:56.702 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-21T11:34:56.702 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:56.702 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T11:34:56.702 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:34:56.702 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:34:56.702 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T11:34:56.703 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T11:34:56.703 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T11:34:56.703 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:56.703 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:56.703 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:56.703 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.b.asok 2026-02-21T11:34:56.703 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:56.703 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.b.asok config get mon_host 2026-02-21T11:34:56.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2026-02-21T11:34:56.835 INFO:tasks.workunit.client.0.trial189.stderr:invalid mon(s) specified: [dne] 2026-02-21T11:34:56.840 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2026-02-21T11:34:56.911 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons [a] 2026-02-21T11:34:56.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2026-02-21T11:34:56.984 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b] 2026-02-21T11:34:56.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2026-02-21T11:34:57.058 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b] 2026-02-21T11:34:57.059 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2026-02-21T11:34:57.153 INFO:tasks.workunit.client.0.trial189.stderr:adding another mon that is not yet online will not break quorum 2026-02-21T11:34:57.159 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2026-02-21T11:34:57.249 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.a 2026-02-21T11:34:57.254 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2026-02-21T11:34:57.348 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.b 2026-02-21T11:34:57.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2026-02-21T11:34:57.447 INFO:tasks.workunit.client.0.trial189.stderr:mon.dne does not exist 2026-02-21T11:34:57.453 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-21T11:34:57.454 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:57.454 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:57.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:57.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:57.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:57.559 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:57.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-21T11:34:57.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T11:34:57.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:34:57.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T11:34:57.560 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:57.560 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:57.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:57.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:57.560 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:57.566 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:57.566 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:34:57.567 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:34:57.567 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:34:57.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:34:57.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:34:57.568 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:34:57.569 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:57.569 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:34:57.569 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:34:57.570 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:57.570 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:34:57.571 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:34:57.571 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T11:34:57.574 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:34:57.574 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:57.574 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:57.574 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.58069 2026-02-21T11:34:57.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:34:57.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:34:57.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-21T11:34:57.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-21T11:34:57.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-21T11:34:57.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-21T11:34:57.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T11:34:57.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:34:57.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T11:34:57.575 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:57.575 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:57.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:57.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:57.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:57.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:57.577 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:34:57.578 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:34:57.578 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:34:57.579 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:34:57.579 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:34:57.579 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:34:57.580 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:57.580 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:34:57.580 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:34:57.581 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:34:57.581 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:34:57.582 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:34:57.582 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T11:34:57.583 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:34:57.583 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:57.583 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:57.583 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.58069 2026-02-21T11:34:57.584 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:34:57.584 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:34:57.584 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-21T11:34:57.585 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:34:57.585 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:57.585 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:57.585 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.58069 2026-02-21T11:34:57.587 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:34:57.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:34:57.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:34:57.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-21T11:34:57.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2026-02-21T11:34:57.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2026-02-21T11:34:57.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2026-02-21T11:34:57.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-21T11:34:57.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:34:57.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:34:57.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:34:57.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:34:57.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-21T11:34:57.587 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-21T11:34:57.604 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:34:57.604 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:57.604 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:57.604 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:57.604 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:57.604 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:57.604 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:57.605 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-21T11:34:57.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:34:57.629 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:34:57.629 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:57.630 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:34:57.630 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:34:57.630 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:34:57.630 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:34:57.630 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:34:57.630 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:34:57.631 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:57.631 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:57.631 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:57.631 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.a.asok 2026-02-21T11:34:57.631 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:57.631 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.a.asok config get fsid 2026-02-21T11:34:57.672 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:34:57.672 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:57.672 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:34:57.672 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:34:57.672 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:34:57.673 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:34:57.673 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:34:57.673 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:34:57.673 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:57.673 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:57.673 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:57.673 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.a.asok 2026-02-21T11:34:57.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:57.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.a.asok config get mon_host 2026-02-21T11:34:57.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-21T11:34:57.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:34:57.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:34:57.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-21T11:34:57.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:34:57.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-21T11:34:57.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-21T11:34:57.730 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:34:57.731 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:57.731 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:57.731 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:57.731 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:57.731 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:57.731 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:57.731 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-21T11:34:57.754 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:34:57.755 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-21T11:34:57.755 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:57.755 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T11:34:57.755 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:34:57.756 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:34:57.756 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T11:34:57.756 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T11:34:57.756 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T11:34:57.756 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:57.756 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:57.756 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:57.757 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.b.asok 2026-02-21T11:34:57.757 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:57.757 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.b.asok config get fsid 2026-02-21T11:34:57.800 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-21T11:34:57.800 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:57.800 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T11:34:57.800 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:34:57.801 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:34:57.801 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T11:34:57.801 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T11:34:57.801 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T11:34:57.801 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:57.801 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:57.801 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:57.801 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.b.asok 2026-02-21T11:34:57.801 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:57.801 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.b.asok config get mon_host 2026-02-21T11:34:57.843 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2026-02-21T11:34:57.843 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:34:57.843 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:34:57.844 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2026-02-21T11:34:57.844 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:34:57.844 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2026-02-21T11:34:57.844 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2026-02-21T11:34:57.859 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:34:57.859 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:34:57.859 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:34:57.859 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:34:57.859 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:57.860 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:57.860 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:34:57.860 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2026-02-21T11:34:57.881 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:34:57.882 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2026-02-21T11:34:57.882 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:57.882 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-21T11:34:57.882 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:34:57.882 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:34:57.882 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-21T11:34:57.882 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-21T11:34:57.882 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-21T11:34:57.883 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:57.883 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:57.883 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:57.883 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.c.asok 2026-02-21T11:34:57.883 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:57.883 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.c.asok config get fsid 2026-02-21T11:34:57.924 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2026-02-21T11:34:57.924 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:34:57.924 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-21T11:34:57.924 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:34:57.924 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:34:57.924 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-21T11:34:57.924 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-21T11:34:57.924 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-21T11:34:57.925 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:34:57.925 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:34:57.925 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:34:57.925 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.c.asok 2026-02-21T11:34:57.925 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:34:57.925 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.c.asok config get mon_host 2026-02-21T11:34:57.967 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2026-02-21T11:34:57.967 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-21T11:34:57.967 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2026-02-21T11:34:57.967 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-21T11:34:57.967 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2026-02-21T11:34:57.968 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-21T11:34:57.968 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-21T11:34:57.968 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771673757 2026-02-21T11:34:57.968 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:34:57.969 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673697 -lt 1771673757 ]] 2026-02-21T11:34:57.969 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:34:57.969 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:34:58.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:34:58.669 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:34:58.669 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:34:58.669 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:34:58.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-21T11:34:58.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-21T11:34:58.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-21T11:34:58.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-21T11:34:58.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-21T11:34:58.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2026-02-21T11:34:58.762 INFO:tasks.workunit.client.0.trial189.stderr:invalid mon(s) specified: [dne] 2026-02-21T11:34:58.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2026-02-21T11:34:58.857 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (b,c) after stopping [a] 2026-02-21T11:34:58.861 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2026-02-21T11:34:58.953 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (a,c) after stopping [b] 2026-02-21T11:34:58.958 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2026-02-21T11:34:59.049 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (a,b) after stopping [c] 2026-02-21T11:34:59.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2026-02-21T11:34:59.128 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a,b] 2026-02-21T11:34:59.130 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2026-02-21T11:34:59.203 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b,c] 2026-02-21T11:34:59.204 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2026-02-21T11:34:59.278 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b,c] 2026-02-21T11:34:59.280 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2026-02-21T11:34:59.372 INFO:tasks.workunit.client.0.trial189.stderr:adding another mon that is not yet online will not break quorum 2026-02-21T11:34:59.378 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2026-02-21T11:34:59.471 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.a 2026-02-21T11:34:59.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2026-02-21T11:34:59.567 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.b 2026-02-21T11:34:59.571 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2026-02-21T11:34:59.666 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.c 2026-02-21T11:34:59.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2026-02-21T11:34:59.674 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:34:59.674 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:34:59.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:34:59.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:34:59.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:34:59.780 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:34:59.780 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2026-02-21T11:34:59.781 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-21T11:34:59.781 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=2 2026-02-21T11:34:59.781 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-21T11:34:59.781 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 2 ]] 2026-02-21T11:34:59.781 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-21T11:34:59.781 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-21T11:34:59.782 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771673759 2026-02-21T11:34:59.782 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:34:59.782 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673699 -lt 1771673759 ]] 2026-02-21T11:34:59.782 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:34:59.783 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:34:59.884 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:34:59.884 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:34:59.885 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:34:59.885 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:34:59.886 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:34:59.886 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:34:59.886 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:34:59.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673699 -lt 1771673759 ]] 2026-02-21T11:34:59.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:34:59.887 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:34:59.987 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:34:59.987 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:34:59.988 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:34:59.988 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:34:59.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:34:59.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:34:59.990 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:34:59.991 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673699 -lt 1771673759 ]] 2026-02-21T11:34:59.991 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:34:59.991 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:00.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:00.089 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:00.090 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:00.090 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:00.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:00.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:00.091 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:00.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673700 -lt 1771673759 ]] 2026-02-21T11:35:00.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:00.093 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:00.193 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:00.194 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:00.194 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:00.194 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:00.196 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:00.196 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:00.196 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:00.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673700 -lt 1771673759 ]] 2026-02-21T11:35:00.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:00.197 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:03.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:03.295 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:03.296 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:03.296 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:03.297 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:03.297 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:03.297 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:03.298 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673703 -lt 1771673759 ]] 2026-02-21T11:35:03.298 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:03.299 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:03.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:03.397 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:03.398 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:03.398 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:03.400 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:03.400 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:03.400 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:03.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673703 -lt 1771673759 ]] 2026-02-21T11:35:03.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:03.401 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:03.503 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:03.504 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:03.504 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:03.505 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:03.506 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:03.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:03.507 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:03.508 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673703 -lt 1771673759 ]] 2026-02-21T11:35:03.508 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:03.508 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:03.607 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:03.607 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:03.607 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:03.608 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:03.609 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:03.610 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:03.610 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:03.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673703 -lt 1771673759 ]] 2026-02-21T11:35:03.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:03.611 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:03.710 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:03.710 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:03.710 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:03.710 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:03.712 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:03.712 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:03.713 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:03.713 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673703 -lt 1771673759 ]] 2026-02-21T11:35:03.713 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:03.714 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:03.811 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:03.811 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:03.811 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:03.812 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:03.813 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:03.813 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:03.814 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:03.815 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673703 -lt 1771673759 ]] 2026-02-21T11:35:03.815 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:03.815 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:03.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:03.914 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:03.914 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:03.914 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:03.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:03.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:03.916 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:03.917 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673703 -lt 1771673759 ]] 2026-02-21T11:35:03.917 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:03.918 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:04.017 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:04.017 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:04.018 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:04.018 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:04.020 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:04.020 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:04.020 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:04.021 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673704 -lt 1771673759 ]] 2026-02-21T11:35:04.021 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:04.021 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:04.119 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:04.119 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:04.119 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:04.119 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:04.121 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:04.121 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:04.122 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:04.122 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673704 -lt 1771673759 ]] 2026-02-21T11:35:04.122 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:04.123 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:04.223 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:04.224 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:04.224 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:04.224 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:04.226 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:04.226 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:04.226 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:04.227 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673704 -lt 1771673759 ]] 2026-02-21T11:35:04.227 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:04.227 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:04.326 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:04.326 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:04.326 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:04.326 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:04.328 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:04.328 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:04.328 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:04.329 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673704 -lt 1771673759 ]] 2026-02-21T11:35:04.329 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:04.329 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:07.428 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:07.428 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:07.428 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:07.428 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:07.431 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:07.431 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:07.431 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:07.432 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673707 -lt 1771673759 ]] 2026-02-21T11:35:07.432 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:07.432 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:07.529 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:07.529 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:07.530 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:07.530 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:07.531 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:07.531 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:07.531 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:07.532 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673707 -lt 1771673759 ]] 2026-02-21T11:35:07.532 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:07.532 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:07.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:07.628 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:07.628 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:07.629 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:07.630 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:07.631 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:07.631 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:07.632 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673707 -lt 1771673759 ]] 2026-02-21T11:35:07.632 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-21T11:35:07.632 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:15.133 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:15.133 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:15.134 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:34:57.598002Z","created":"2026-02-21T11:34:57.598002Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:597997+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598000+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:34:57:598001+0000","crush_location":[]}]}}' 2026-02-21T11:35:15.134 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-21T11:35:15.136 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-21T11:35:15.136 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-21T11:35:15.136 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-21T11:35:15.136 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-21T11:35:15.136 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-21T11:35:15.136 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2026-02-21T11:35:15.211 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a] 2026-02-21T11:35:15.214 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2026-02-21T11:35:15.304 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (a,c) after stopping [b] 2026-02-21T11:35:15.309 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2026-02-21T11:35:18.381 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [c] 2026-02-21T11:35:18.383 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2026-02-21T11:35:18.457 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2026-02-21T11:35:18.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2026-02-21T11:35:21.533 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: removing mon.a would break quorum 2026-02-21T11:35:21.535 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2026-02-21T11:35:21.627 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.b 2026-02-21T11:35:21.632 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2026-02-21T11:35:21.704 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: removing mon.c would break quorum 2026-02-21T11:35:21.706 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-21T11:35:21.706 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:35:21.706 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:35:21.706 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:35:21.706 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:35:21.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:35:21.813 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:35:21.813 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-21T11:35:21.813 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T11:35:21.813 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:35:21.813 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T11:35:21.813 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:35:21.813 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:35:21.813 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:35:21.813 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:35:21.813 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:35:21.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:35:21.819 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:35:21.820 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:35:21.820 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:35:21.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:35:21.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:35:21.821 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:35:21.822 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:35:21.822 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:35:21.823 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:35:21.823 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:35:21.823 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:35:21.824 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:35:21.824 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T11:35:21.829 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:35:21.829 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:21.829 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:21.830 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.58069 2026-02-21T11:35:21.831 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:35:21.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:35:21.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-21T11:35:21.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-21T11:35:21.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-21T11:35:21.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-21T11:35:21.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T11:35:21.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:35:21.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T11:35:21.832 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:35:21.832 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:35:21.833 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:35:21.833 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:35:21.833 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:35:21.834 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:35:21.834 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:35:21.834 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:35:21.834 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:35:21.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:35:21.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:35:21.835 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:35:21.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:35:21.836 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:35:21.836 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:35:21.836 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:35:21.837 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:35:21.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:35:21.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T11:35:21.839 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:35:21.839 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:21.839 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:21.839 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.58069 2026-02-21T11:35:21.840 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:35:21.840 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:35:21.840 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-21T11:35:21.841 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:35:21.841 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:21.841 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:21.841 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.58069 2026-02-21T11:35:21.842 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:35:21.844 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:35:21.844 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:35:21.844 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-21T11:35:21.844 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2026-02-21T11:35:21.844 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2026-02-21T11:35:21.844 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2026-02-21T11:35:21.844 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-21T11:35:21.844 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:35:21.845 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:35:21.845 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:35:21.845 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:35:21.845 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-21T11:35:21.845 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-21T11:35:21.861 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:35:21.861 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:35:21.861 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:35:21.861 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:35:21.861 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:21.861 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:21.861 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:35:21.861 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-21T11:35:21.884 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:35:21.886 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:35:21.887 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:21.887 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:35:21.887 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:35:21.887 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:35:21.887 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:35:21.887 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:35:21.887 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:35:21.887 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:21.887 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:21.887 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:21.887 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.a.asok 2026-02-21T11:35:21.887 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:21.888 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.a.asok config get fsid 2026-02-21T11:35:21.932 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:35:21.932 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:21.932 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:35:21.932 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:35:21.932 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:35:21.932 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:35:21.932 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:35:21.933 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:35:21.933 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:21.933 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:21.933 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:21.933 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.a.asok 2026-02-21T11:35:21.933 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:21.933 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.a.asok config get mon_host 2026-02-21T11:35:21.974 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-21T11:35:21.974 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:35:21.974 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:35:21.974 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-21T11:35:21.974 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:35:21.974 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-21T11:35:21.974 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-21T11:35:21.992 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:35:21.992 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:35:21.992 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:35:21.992 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:35:21.992 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:21.992 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:21.992 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:35:21.992 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-21T11:35:22.015 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:35:22.016 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-21T11:35:22.016 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:22.017 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T11:35:22.017 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:35:22.017 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:35:22.018 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T11:35:22.018 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T11:35:22.018 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T11:35:22.018 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:22.018 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:22.019 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:22.019 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.b.asok 2026-02-21T11:35:22.019 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:22.019 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.b.asok config get fsid 2026-02-21T11:35:22.061 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-21T11:35:22.061 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:22.061 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T11:35:22.061 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:35:22.062 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:35:22.062 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T11:35:22.062 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T11:35:22.062 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T11:35:22.062 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:22.062 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:22.062 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:22.062 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.b.asok 2026-02-21T11:35:22.063 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:22.063 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.b.asok config get mon_host 2026-02-21T11:35:22.106 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2026-02-21T11:35:22.106 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:35:22.106 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:35:22.107 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2026-02-21T11:35:22.107 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:35:22.107 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2026-02-21T11:35:22.107 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2026-02-21T11:35:22.122 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:35:22.122 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:35:22.122 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:35:22.122 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:35:22.122 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:22.122 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:22.123 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:35:22.123 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2026-02-21T11:35:22.146 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:35:22.147 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2026-02-21T11:35:22.148 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:22.148 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-21T11:35:22.148 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:35:22.148 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:35:22.148 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-21T11:35:22.148 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-21T11:35:22.148 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-21T11:35:22.148 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:22.148 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:22.148 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:22.149 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.c.asok 2026-02-21T11:35:22.149 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:22.149 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.c.asok config get fsid 2026-02-21T11:35:22.191 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2026-02-21T11:35:22.191 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:22.191 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-21T11:35:22.191 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:35:22.191 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:35:22.191 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-21T11:35:22.192 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-21T11:35:22.192 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-21T11:35:22.192 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:22.192 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:22.192 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:22.192 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.c.asok 2026-02-21T11:35:22.192 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:22.192 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.c.asok config get mon_host 2026-02-21T11:35:22.233 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2026-02-21T11:35:22.233 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:35:22.233 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:35:22.233 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2026-02-21T11:35:22.233 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:35:22.233 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/d 2026-02-21T11:35:22.233 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2026-02-21T11:35:22.249 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:35:22.249 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:35:22.249 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:35:22.250 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:35:22.250 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:22.250 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:22.250 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:35:22.250 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2026-02-21T11:35:22.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:35:22.275 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2026-02-21T11:35:22.275 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:22.275 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-21T11:35:22.275 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:35:22.275 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:35:22.275 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-21T11:35:22.275 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-21T11:35:22.275 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-21T11:35:22.276 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:22.276 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:22.276 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:22.276 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.d.asok 2026-02-21T11:35:22.276 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:22.277 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.d.asok config get fsid 2026-02-21T11:35:22.321 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2026-02-21T11:35:22.321 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:22.322 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-21T11:35:22.322 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:35:22.322 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:35:22.322 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-21T11:35:22.322 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-21T11:35:22.322 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-21T11:35:22.322 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:22.322 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:22.322 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:22.322 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.d.asok 2026-02-21T11:35:22.322 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:22.322 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.d.asok config get mon_host 2026-02-21T11:35:22.363 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2026-02-21T11:35:22.364 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-21T11:35:22.364 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2026-02-21T11:35:22.364 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-21T11:35:22.364 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2026-02-21T11:35:22.364 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-21T11:35:22.364 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-21T11:35:22.365 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771673782 2026-02-21T11:35:22.365 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:22.365 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673722 -lt 1771673782 ]] 2026-02-21T11:35:22.365 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:35:22.365 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:25.466 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:25.466 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:25.467 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:35:25.467 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:25.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-21T11:35:25.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-21T11:35:25.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-21T11:35:25.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-21T11:35:25.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-21T11:35:25.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2026-02-21T11:35:25.561 INFO:tasks.workunit.client.0.trial189.stderr:invalid mon(s) specified: [dne] 2026-02-21T11:35:25.566 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2026-02-21T11:35:28.660 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (b,c,d) after stopping [a] 2026-02-21T11:35:28.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2026-02-21T11:35:28.757 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (a,c,d) after stopping [b] 2026-02-21T11:35:28.762 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2026-02-21T11:35:28.854 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (a,b,d) after stopping [c] 2026-02-21T11:35:28.859 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2026-02-21T11:35:28.953 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (a,b,c) after stopping [d] 2026-02-21T11:35:28.958 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2026-02-21T11:35:29.033 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [a,b] 2026-02-21T11:35:29.035 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2026-02-21T11:35:32.110 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons [c,d] 2026-02-21T11:35:32.112 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2026-02-21T11:35:32.205 INFO:tasks.workunit.client.0.trial189.stderr:adding another mon that is not yet online will not break quorum 2026-02-21T11:35:32.211 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2026-02-21T11:35:32.304 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.a 2026-02-21T11:35:32.309 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2026-02-21T11:35:32.401 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.b 2026-02-21T11:35:32.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2026-02-21T11:35:32.498 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.c 2026-02-21T11:35:32.503 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2026-02-21T11:35:32.504 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:35:32.504 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:35:32.504 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:35:32.504 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:35:32.504 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:35:32.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:35:32.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2026-02-21T11:35:32.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-21T11:35:32.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2026-02-21T11:35:32.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-21T11:35:32.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2026-02-21T11:35:32.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-21T11:35:32.611 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-21T11:35:32.612 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771673792 2026-02-21T11:35:32.612 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:32.613 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673732 -lt 1771673792 ]] 2026-02-21T11:35:32.613 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:35:32.613 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:32.713 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:32.714 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:32.714 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:32.714 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:35:32.716 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:32.716 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:32.716 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:32.717 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673732 -lt 1771673792 ]] 2026-02-21T11:35:32.717 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:35:32.717 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:35.822 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:35.822 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:35.823 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:35.823 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:35:35.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:35.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:35.825 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:35.826 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673735 -lt 1771673792 ]] 2026-02-21T11:35:35.826 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:35:35.826 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:35.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:35.929 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:35.929 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:35.929 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:35:35.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:35.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:35.931 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:35.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673735 -lt 1771673792 ]] 2026-02-21T11:35:35.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:35:35.932 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:36.034 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:36.034 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:36.035 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:36.035 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:35:36.037 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:36.037 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:36.037 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:36.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673736 -lt 1771673792 ]] 2026-02-21T11:35:36.039 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:35:36.039 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:36.142 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:36.142 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:36.143 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:36.143 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:35:36.144 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:36.144 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:36.144 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:36.145 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673736 -lt 1771673792 ]] 2026-02-21T11:35:36.145 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:35:36.146 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:36.244 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:36.244 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:36.245 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:36.245 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:35:36.247 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:36.247 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:36.247 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:36.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673736 -lt 1771673792 ]] 2026-02-21T11:35:36.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:35:36.249 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:36.346 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:36.346 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:36.346 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:36.346 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:35:36.348 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:36.348 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:36.348 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:36.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673736 -lt 1771673792 ]] 2026-02-21T11:35:36.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:35:36.350 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:36.448 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:36.448 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:36.449 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:36.449 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:35:36.450 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:36.450 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:36.450 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:36.451 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673736 -lt 1771673792 ]] 2026-02-21T11:35:36.451 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:35:36.451 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:36.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:36.549 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:36.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:35:36.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:36.551 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:36.551 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:36.551 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:36.552 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673736 -lt 1771673792 ]] 2026-02-21T11:35:36.552 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:35:36.553 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:39.650 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:39.650 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:39.651 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:39.651 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:35:39.653 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:39.653 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:39.653 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:39.654 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673739 -lt 1771673792 ]] 2026-02-21T11:35:39.654 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:35:39.654 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:39.751 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:39.751 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:39.752 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:39.752 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:35:39.754 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:39.754 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:39.754 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:39.755 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673739 -lt 1771673792 ]] 2026-02-21T11:35:39.755 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:35:39.755 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:39.854 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:39.854 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:39.855 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:39.855 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:35:39.856 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:39.856 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:39.856 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:39.857 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673739 -lt 1771673792 ]] 2026-02-21T11:35:39.857 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:35:39.857 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:39.958 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:39.959 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:39.959 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:39.960 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:35:39.961 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:35:39.961 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:35:39.961 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:39.962 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673739 -lt 1771673792 ]] 2026-02-21T11:35:39.962 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:35:39.962 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:49.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:49.062 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":6,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:49.062 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:21.853673Z","created":"2026-02-21T11:35:21.853673Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853667+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853670+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853671+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:21:853672+0000","crush_location":[]}]}}' 2026-02-21T11:35:49.062 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:35:49.064 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-21T11:35:49.064 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-21T11:35:49.064 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-21T11:35:49.064 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-21T11:35:49.064 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-21T11:35:49.064 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2026-02-21T11:35:49.156 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (b,c,d) after stopping [a] 2026-02-21T11:35:49.162 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2026-02-21T11:35:49.232 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2026-02-21T11:35:49.234 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2026-02-21T11:35:49.309 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2026-02-21T11:35:49.311 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2026-02-21T11:35:49.383 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2026-02-21T11:35:49.385 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2026-02-21T11:35:49.477 INFO:tasks.workunit.client.0.trial189.stderr:adding another mon that is not yet online will not break quorum 2026-02-21T11:35:49.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2026-02-21T11:35:49.577 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.a 2026-02-21T11:35:49.582 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2026-02-21T11:35:49.672 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.b 2026-02-21T11:35:49.677 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2026-02-21T11:35:49.769 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.c 2026-02-21T11:35:49.774 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2026-02-21T11:35:49.865 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.d 2026-02-21T11:35:49.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-21T11:35:49.870 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:35:49.870 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:35:49.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:35:49.871 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:35:49.871 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:35:49.977 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:35:49.977 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-21T11:35:49.977 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T11:35:49.977 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:35:49.977 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T11:35:49.977 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:35:49.978 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:35:49.978 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:35:49.978 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:35:49.978 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:35:49.984 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:35:49.985 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:35:49.985 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:35:49.986 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:35:49.987 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:35:49.987 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:35:49.987 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:35:49.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:35:49.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:35:49.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:35:49.988 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:35:49.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:35:49.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:35:49.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T11:35:49.996 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:35:49.996 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:49.996 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:49.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.58069 2026-02-21T11:35:49.998 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:35:49.998 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:35:49.998 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-21T11:35:49.998 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-21T11:35:49.998 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-21T11:35:49.998 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-21T11:35:49.999 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T11:35:49.999 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:35:49.999 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T11:35:49.999 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:35:49.999 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:35:49.999 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:35:49.999 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:35:49.999 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:35:50.000 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:35:50.000 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:35:50.000 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:35:50.000 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:35:50.002 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:35:50.002 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:35:50.002 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:35:50.003 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:35:50.003 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:35:50.003 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:35:50.003 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:35:50.004 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:35:50.005 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:35:50.005 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T11:35:50.006 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:35:50.006 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.006 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.006 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.58069 2026-02-21T11:35:50.007 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:35:50.007 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:35:50.007 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-21T11:35:50.008 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:35:50.008 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.008 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.58069 2026-02-21T11:35:50.009 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:35:50.011 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:35:50.011 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:35:50.011 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-21T11:35:50.011 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2026-02-21T11:35:50.012 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2026-02-21T11:35:50.012 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=0c8fbdc5-4188-4947-9f19-3787b3c1cc5a --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2026-02-21T11:35:50.012 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-21T11:35:50.012 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:35:50.012 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:35:50.012 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:35:50.012 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:35:50.012 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-21T11:35:50.012 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-21T11:35:50.029 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:35:50.029 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:35:50.029 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:35:50.029 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:35:50.029 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.029 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.029 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:35:50.029 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-21T11:35:50.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:35:50.055 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:35:50.055 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:50.055 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:35:50.055 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:35:50.055 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:35:50.055 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:35:50.055 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:35:50.055 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:35:50.055 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:50.056 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.056 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.056 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.a.asok 2026-02-21T11:35:50.056 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:50.056 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.a.asok config get fsid 2026-02-21T11:35:50.100 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:35:50.100 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:50.100 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:35:50.100 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:35:50.100 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:35:50.100 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:35:50.101 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:35:50.101 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:35:50.101 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:50.101 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.101 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.101 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.a.asok 2026-02-21T11:35:50.101 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:50.101 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.a.asok config get mon_host 2026-02-21T11:35:50.143 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-21T11:35:50.143 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:35:50.143 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:35:50.143 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-21T11:35:50.143 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:35:50.143 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-21T11:35:50.143 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-21T11:35:50.160 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:35:50.160 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:35:50.160 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:35:50.160 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:35:50.160 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.160 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.161 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:35:50.161 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-21T11:35:50.184 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:35:50.185 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-21T11:35:50.185 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:50.185 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T11:35:50.185 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:35:50.185 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:35:50.186 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T11:35:50.186 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T11:35:50.186 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T11:35:50.187 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:50.187 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.188 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.188 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.b.asok 2026-02-21T11:35:50.188 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:50.188 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.b.asok config get fsid 2026-02-21T11:35:50.231 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-21T11:35:50.231 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:50.231 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-21T11:35:50.231 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:35:50.231 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:35:50.231 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-21T11:35:50.231 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-21T11:35:50.231 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-21T11:35:50.231 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:50.231 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.231 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.231 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.b.asok 2026-02-21T11:35:50.232 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:50.232 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.b.asok config get mon_host 2026-02-21T11:35:50.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2026-02-21T11:35:50.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:35:50.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:35:50.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2026-02-21T11:35:50.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:35:50.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2026-02-21T11:35:50.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2026-02-21T11:35:50.291 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:35:50.292 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:35:50.292 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:35:50.292 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:35:50.292 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.292 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.292 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:35:50.292 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2026-02-21T11:35:50.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:35:50.319 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2026-02-21T11:35:50.319 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:50.319 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-21T11:35:50.319 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:35:50.320 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:35:50.320 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-21T11:35:50.320 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-21T11:35:50.320 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-21T11:35:50.320 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:50.320 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.321 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.321 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.c.asok 2026-02-21T11:35:50.321 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:50.321 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.c.asok config get fsid 2026-02-21T11:35:50.359 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2026-02-21T11:35:50.359 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:50.359 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-21T11:35:50.359 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:35:50.360 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:35:50.360 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-21T11:35:50.360 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-21T11:35:50.360 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-21T11:35:50.360 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:50.360 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.360 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.360 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.c.asok 2026-02-21T11:35:50.360 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:50.360 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.c.asok config get mon_host 2026-02-21T11:35:50.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2026-02-21T11:35:50.403 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:35:50.403 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:35:50.403 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2026-02-21T11:35:50.403 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:35:50.403 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/d 2026-02-21T11:35:50.403 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2026-02-21T11:35:50.419 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:35:50.419 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:35:50.419 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:35:50.419 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:35:50.419 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.419 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.420 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:35:50.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2026-02-21T11:35:50.443 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:35:50.446 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2026-02-21T11:35:50.447 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:50.447 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-21T11:35:50.447 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:35:50.447 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:35:50.447 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-21T11:35:50.448 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-21T11:35:50.448 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-21T11:35:50.448 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:50.448 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.448 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.448 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.d.asok 2026-02-21T11:35:50.448 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:50.449 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.d.asok config get fsid 2026-02-21T11:35:50.490 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2026-02-21T11:35:50.491 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:50.491 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-21T11:35:50.491 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:35:50.491 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:35:50.491 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-21T11:35:50.491 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-21T11:35:50.491 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-21T11:35:50.491 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:50.492 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.492 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.492 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.d.asok 2026-02-21T11:35:50.492 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:50.492 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.d.asok config get mon_host 2026-02-21T11:35:50.533 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2026-02-21T11:35:50.534 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-21T11:35:50.534 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:35:50.534 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=e 2026-02-21T11:35:50.534 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:35:50.534 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/e 2026-02-21T11:35:50.534 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2026-02-21T11:35:50.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:35:50.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:35:50.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:35:50.550 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:35:50.550 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.550 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.58069/$cluster-$name.asok' 2026-02-21T11:35:50.551 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.58069/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2026-02-21T11:35:50.573 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:35:50.573 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e fsid 2026-02-21T11:35:50.574 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:50.574 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2026-02-21T11:35:50.574 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:35:50.574 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:35:50.575 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2026-02-21T11:35:50.575 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2026-02-21T11:35:50.575 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2026-02-21T11:35:50.575 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:50.575 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.576 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.576 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.e.asok 2026-02-21T11:35:50.576 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:50.576 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.e.asok config get fsid 2026-02-21T11:35:50.617 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e mon_host 2026-02-21T11:35:50.617 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:35:50.617 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2026-02-21T11:35:50.617 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:35:50.617 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:35:50.617 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2026-02-21T11:35:50.618 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2026-02-21T11:35:50.618 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2026-02-21T11:35:50.618 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:35:50.618 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:35:50.618 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:35:50.618 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.58069/ceph-mon.e.asok 2026-02-21T11:35:50.618 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:35:50.618 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.58069/ceph-mon.e.asok config get mon_host 2026-02-21T11:35:50.660 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2026-02-21T11:35:50.660 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-21T11:35:50.660 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=5 2026-02-21T11:35:50.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-21T11:35:50.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 5 ]] 2026-02-21T11:35:50.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-21T11:35:50.661 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-21T11:35:50.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771673810 2026-02-21T11:35:50.661 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:35:50.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673750 -lt 1771673810 ]] 2026-02-21T11:35:50.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 5' 2026-02-21T11:35:50.662 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:35:56.764 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:35:56.764 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:35:56.765 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:35:56.765 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 5' 2026-02-21T11:35:56.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-21T11:35:56.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-21T11:35:56.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-21T11:35:56.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-21T11:35:56.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-21T11:35:56.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2026-02-21T11:35:56.859 INFO:tasks.workunit.client.0.trial189.stderr:invalid mon(s) specified: [dne] 2026-02-21T11:35:56.864 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2026-02-21T11:35:56.956 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2026-02-21T11:35:56.962 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2026-02-21T11:35:57.056 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (a,c,d,e) after stopping [b] 2026-02-21T11:35:57.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2026-02-21T11:35:57.154 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (a,b,d,e) after stopping [c] 2026-02-21T11:35:57.159 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2026-02-21T11:35:57.250 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (a,b,c,e) after stopping [d] 2026-02-21T11:35:57.255 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2026-02-21T11:35:57.348 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (a,b,c,d) after stopping [e] 2026-02-21T11:35:57.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2026-02-21T11:35:57.449 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (c,d,e) after stopping [a,b] 2026-02-21T11:35:57.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2026-02-21T11:35:57.549 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (a,b,e) after stopping [c,d] 2026-02-21T11:35:57.554 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2026-02-21T11:35:57.631 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons [a,b,c] 2026-02-21T11:35:57.632 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2026-02-21T11:35:57.729 INFO:tasks.workunit.client.0.trial189.stderr:adding another mon that is not yet online will not break quorum 2026-02-21T11:35:57.734 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2026-02-21T11:35:57.829 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.a 2026-02-21T11:35:57.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2026-02-21T11:35:57.929 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.b 2026-02-21T11:35:57.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2026-02-21T11:35:58.030 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.c 2026-02-21T11:35:58.036 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2026-02-21T11:36:01.129 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.d 2026-02-21T11:36:01.134 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2026-02-21T11:36:01.229 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.e 2026-02-21T11:36:01.235 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2026-02-21T11:36:01.235 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:36:01.235 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:36:01.235 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:36:01.235 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:36:01.235 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:36:01.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:36:01.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2026-02-21T11:36:01.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-21T11:36:01.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2026-02-21T11:36:01.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-21T11:36:01.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2026-02-21T11:36:01.344 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-21T11:36:01.344 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-21T11:36:01.344 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771673821 2026-02-21T11:36:01.344 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:01.345 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673761 -lt 1771673821 ]] 2026-02-21T11:36:01.345 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:01.346 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:04.445 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:04.445 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:04.446 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:04.446 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:04.448 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:04.448 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:04.448 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:04.449 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673764 -lt 1771673821 ]] 2026-02-21T11:36:04.449 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:04.449 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:04.548 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:04.548 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:04.549 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:04.549 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:04.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:04.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:04.551 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:04.551 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673764 -lt 1771673821 ]] 2026-02-21T11:36:04.551 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:04.552 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:04.651 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:04.651 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:04.652 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:04.652 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:04.654 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:04.654 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:04.654 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:04.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673764 -lt 1771673821 ]] 2026-02-21T11:36:04.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:04.655 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:04.753 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:04.753 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:04.754 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:04.754 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:04.755 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:04.755 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:04.756 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:04.756 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673764 -lt 1771673821 ]] 2026-02-21T11:36:04.757 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:04.757 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:04.855 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:04.856 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:04.856 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:04.857 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:04.858 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:04.858 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:04.858 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:04.859 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673764 -lt 1771673821 ]] 2026-02-21T11:36:04.859 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:04.859 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:04.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:04.959 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:04.959 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:04.959 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:04.961 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:04.961 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:04.962 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:04.962 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673764 -lt 1771673821 ]] 2026-02-21T11:36:04.962 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:04.962 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:05.059 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:05.059 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:05.059 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:05.059 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:05.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:05.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:05.062 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:05.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673765 -lt 1771673821 ]] 2026-02-21T11:36:05.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:05.063 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:05.160 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:05.160 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:05.160 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:05.160 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:05.163 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:05.163 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:05.163 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:05.164 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673765 -lt 1771673821 ]] 2026-02-21T11:36:05.164 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:05.164 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:05.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:05.265 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:05.266 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:05.266 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:05.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:05.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:05.268 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:05.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673765 -lt 1771673821 ]] 2026-02-21T11:36:05.269 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:05.269 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:05.370 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:05.370 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:05.371 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:05.371 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:05.372 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:05.373 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:05.373 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:05.374 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673765 -lt 1771673821 ]] 2026-02-21T11:36:05.374 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:05.374 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:05.471 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:05.471 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:05.471 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:05.472 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:05.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:05.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:05.474 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:05.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673765 -lt 1771673821 ]] 2026-02-21T11:36:05.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:05.475 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:08.574 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:08.574 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:08.575 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:08.575 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:08.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:08.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:08.577 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:08.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673768 -lt 1771673821 ]] 2026-02-21T11:36:08.578 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:08.578 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:08.675 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:08.675 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:08.676 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:08.676 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:08.678 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:08.678 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:08.678 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:08.679 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673768 -lt 1771673821 ]] 2026-02-21T11:36:08.679 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:08.679 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:08.781 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:08.781 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:08.782 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:08.782 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:08.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:08.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:08.783 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:08.784 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673768 -lt 1771673821 ]] 2026-02-21T11:36:08.784 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:08.784 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:08.885 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:08.885 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:08.885 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:08.885 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:08.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:08.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:08.888 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:08.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673768 -lt 1771673821 ]] 2026-02-21T11:36:08.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:08.889 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:08.987 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:08.987 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:08.988 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:08.988 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:08.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:08.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:08.989 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:08.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673768 -lt 1771673821 ]] 2026-02-21T11:36:08.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-21T11:36:08.990 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:16.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:16.496 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:16.496 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:16.497 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-21T11:36:16.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-21T11:36:16.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-21T11:36:16.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-21T11:36:16.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-21T11:36:16.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-21T11:36:16.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2026-02-21T11:36:19.592 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2026-02-21T11:36:19.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2026-02-21T11:36:19.695 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (c,d,e) after stopping [b] 2026-02-21T11:36:19.701 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2026-02-21T11:36:19.794 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (b,d,e) after stopping [c] 2026-02-21T11:36:19.799 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2026-02-21T11:36:19.892 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (b,c,e) after stopping [d] 2026-02-21T11:36:19.897 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2026-02-21T11:36:19.990 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (b,c,d) after stopping [e] 2026-02-21T11:36:19.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2026-02-21T11:36:20.087 INFO:tasks.workunit.client.0.trial189.stderr:adding another mon that is not yet online will not break quorum 2026-02-21T11:36:20.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2026-02-21T11:36:20.184 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.a 2026-02-21T11:36:20.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2026-02-21T11:36:23.283 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.b 2026-02-21T11:36:23.288 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2026-02-21T11:36:23.380 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.c 2026-02-21T11:36:23.385 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2026-02-21T11:36:23.480 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.d 2026-02-21T11:36:23.486 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2026-02-21T11:36:23.582 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.e 2026-02-21T11:36:23.588 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2026-02-21T11:36:23.588 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:36:23.588 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:36:23.589 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:36:23.589 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:36:23.589 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:36:23.695 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:36:23.695 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2026-02-21T11:36:23.695 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-21T11:36:23.695 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2026-02-21T11:36:23.695 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-21T11:36:23.695 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2026-02-21T11:36:23.696 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-21T11:36:23.696 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-21T11:36:23.697 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771673843 2026-02-21T11:36:23.697 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:23.697 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673783 -lt 1771673843 ]] 2026-02-21T11:36:23.697 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:36:23.697 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:23.798 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:23.799 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:23.799 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:23.800 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:36:23.801 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:23.801 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:23.801 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:23.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673783 -lt 1771673843 ]] 2026-02-21T11:36:23.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:36:23.802 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:23.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:23.904 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:23.905 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:36:23.905 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:23.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:23.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:23.907 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:23.908 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673783 -lt 1771673843 ]] 2026-02-21T11:36:23.908 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:36:23.908 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:24.005 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:24.005 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:24.006 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:24.006 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:36:24.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:24.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:24.008 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:24.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673784 -lt 1771673843 ]] 2026-02-21T11:36:24.009 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:36:24.009 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:24.106 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:24.106 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:24.107 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":7,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:24.107 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:36:24.108 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:24.108 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:24.109 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:24.109 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673784 -lt 1771673843 ]] 2026-02-21T11:36:24.109 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:36:24.110 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:27.208 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:27.208 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:27.209 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:27.209 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:36:27.210 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:27.211 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:27.211 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:27.212 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673787 -lt 1771673843 ]] 2026-02-21T11:36:27.212 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:36:27.212 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:30.311 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:30.312 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:30.312 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:36:30.313 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:30.314 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:30.314 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:30.314 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:30.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673790 -lt 1771673843 ]] 2026-02-21T11:36:30.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:36:30.316 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:30.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:30.413 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:30.414 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:30.414 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:36:30.415 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:30.415 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:30.416 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:30.416 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673790 -lt 1771673843 ]] 2026-02-21T11:36:30.416 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:36:30.416 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:33.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:33.517 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":17,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:33.518 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":17,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:33.518 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:36:33.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-21T11:36:33.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-21T11:36:33.520 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-21T11:36:33.521 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771673793 -lt 1771673843 ]] 2026-02-21T11:36:33.521 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-21T11:36:33.521 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-21T11:36:39.619 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-21T11:36:39.620 INFO:tasks.workunit.client.0.trial189.stderr:{"election_epoch":14,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:39.620 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":14,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"0c8fbdc5-4188-4947-9f19-3787b3c1cc5a","modified":"2026-02-21T11:35:50.021327Z","created":"2026-02-21T11:35:50.021327Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021317+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021321+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021323+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021324+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-21T11:35:50:021325+0000","crush_location":[]}]}}' 2026-02-21T11:36:39.620 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-21T11:36:39.622 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-21T11:36:39.622 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-21T11:36:39.622 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-21T11:36:39.622 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-21T11:36:39.622 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-21T11:36:39.622 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2026-02-21T11:36:39.713 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (b,c,d) after stopping [a] 2026-02-21T11:36:39.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2026-02-21T11:36:39.791 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2026-02-21T11:36:39.793 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2026-02-21T11:36:39.867 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2026-02-21T11:36:39.869 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2026-02-21T11:36:42.941 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2026-02-21T11:36:42.943 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2026-02-21T11:36:46.036 INFO:tasks.workunit.client.0.trial189.stderr:quorum should be preserved (b,c,d) after stopping [e] 2026-02-21T11:36:46.041 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2026-02-21T11:36:46.114 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2026-02-21T11:36:46.116 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2026-02-21T11:36:52.208 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.a 2026-02-21T11:36:52.214 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2026-02-21T11:36:55.288 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: removing mon.b would break quorum 2026-02-21T11:36:55.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2026-02-21T11:36:55.365 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: removing mon.c would break quorum 2026-02-21T11:36:55.367 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2026-02-21T11:36:55.442 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: removing mon.d would break quorum 2026-02-21T11:36:55.445 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2026-02-21T11:36:58.541 INFO:tasks.workunit.client.0.trial189.stderr:safe to remove mon.e 2026-02-21T11:36:58.547 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-21T11:36:58.548 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:36:58.548 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:36:58.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:36:58.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:36:58.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:36:58.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:36:58.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-21T11:36:58.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T11:36:58.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:36:58.657 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T11:36:58.657 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:36:58.657 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:36:58.657 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:36:58.657 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:36:58.657 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:36:58.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:36:58.662 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:36:58.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:36:58.663 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:36:58.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:36:58.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:36:58.664 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:36:58.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:36:58.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:36:58.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:36:58.665 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:36:58.666 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:36:58.667 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:36:58.667 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T11:36:58.677 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:36:58.677 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:36:58.677 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:36:58.677 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.58069 2026-02-21T11:36:58.679 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:36:58.679 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:36:58.679 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: main: code=0 2026-02-21T11:36:58.679 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2433: main: teardown td/ok-to-stop 0 2026-02-21T11:36:58.679 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-21T11:36:58.679 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-21T11:36:58.679 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-21T11:36:58.680 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:36:58.680 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:36:58.680 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:36:58.680 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:36:58.680 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:36:58.680 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:36:58.681 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:36:58.681 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:36:58.682 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:36:58.683 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:36:58.683 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:36:58.683 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:36:58.684 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:36:58.684 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:36:58.684 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:36:58.684 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:36:58.685 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:36:58.686 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-21T11:36:58.686 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-21T11:36:58.687 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:36:58.687 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:36:58.687 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.58069 2026-02-21T11:36:58.687 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.58069 2026-02-21T11:36:58.689 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:36:58.689 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:36:58.689 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: main: return 0 2026-02-21T11:36:58.690 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-21T11:36:58.690 DEBUG:teuthology.orchestra.run.trial189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-21T11:36:58.754 INFO:tasks.workunit:Running workunit misc/ok-to-upgrade.sh... 2026-02-21T11:36:58.754 DEBUG:teuthology.orchestra.run.trial189:workunit test misc/ok-to-upgrade.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh 2026-02-21T11:36:58.816 INFO:tasks.workunit.client.0.trial189.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-21T11:36:58.820 INFO:tasks.workunit.client.0.trial189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-21T11:36:58.820 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-21T11:36:58.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-21T11:36:58.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: main: export PYTHONWARNINGS=ignore 2026-02-21T11:36:58.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: main: PYTHONWARNINGS=ignore 2026-02-21T11:36:58.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: main: export CEPH_CONF=/dev/null 2026-02-21T11:36:58.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: main: CEPH_CONF=/dev/null 2026-02-21T11:36:58.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: main: unset CEPH_ARGS 2026-02-21T11:36:58.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: main: local code 2026-02-21T11:36:58.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: main: run td/ok-to-upgrade 2026-02-21T11:36:58.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:21: run: local dir=td/ok-to-upgrade 2026-02-21T11:36:58.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:22: run: shift 2026-02-21T11:36:58.822 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:24: run: export CEPH_MON=127.0.0.1:7170 2026-02-21T11:36:58.822 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:24: run: CEPH_MON=127.0.0.1:7170 2026-02-21T11:36:58.822 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:25: run: export CEPH_ARGS 2026-02-21T11:36:58.822 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:26: run: uuidgen 2026-02-21T11:36:58.822 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:26: run: CEPH_ARGS+='--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none ' 2026-02-21T11:36:58.822 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:27: run: export 'ORIG_CEPH_ARGS=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none ' 2026-02-21T11:36:58.822 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:27: run: ORIG_CEPH_ARGS='--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none ' 2026-02-21T11:36:58.822 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:29: run: set 2026-02-21T11:36:58.822 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:29: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-21T11:36:58.824 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:29: run: local 'funcs=TEST_ok_to_upgrade_bad_osd_version 2026-02-21T11:36:58.824 INFO:tasks.workunit.client.0.trial189.stderr:TEST_ok_to_upgrade_erasure_pool 2026-02-21T11:36:58.824 INFO:tasks.workunit.client.0.trial189.stderr:TEST_ok_to_upgrade_invalid_args 2026-02-21T11:36:58.824 INFO:tasks.workunit.client.0.trial189.stderr:TEST_ok_to_upgrade_replicated_pool' 2026-02-21T11:36:58.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:30: run: for func in $funcs 2026-02-21T11:36:58.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:31: run: setup td/ok-to-upgrade 2026-02-21T11:36:58.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-upgrade 2026-02-21T11:36:58.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-upgrade 2026-02-21T11:36:58.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-upgrade 2026-02-21T11:36:58.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:36:58.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-upgrade KILL 2026-02-21T11:36:58.825 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:36:58.825 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:36:58.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:36:58.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:36:58.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:36:58.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:36:58.826 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:36:58.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:36:58.827 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:36:58.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:36:58.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:36:58.827 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:36:58.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:36:58.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:36:58.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:36:58.828 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:36:58.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:36:58.829 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:36:58.829 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-upgrade 2026-02-21T11:36:58.830 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:36:58.830 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:36:58.830 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:36:58.830 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73624 2026-02-21T11:36:58.831 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:36:58.831 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:36:58.831 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-upgrade 2026-02-21T11:36:58.832 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:36:58.832 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:36:58.832 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:36:58.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73624 2026-02-21T11:36:58.833 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:36:58.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-21T11:36:58.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-21T11:36:58.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:36:58.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-upgrade 1' TERM HUP INT 2026-02-21T11:36:58.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:32: run: TEST_ok_to_upgrade_bad_osd_version td/ok-to-upgrade 2026-02-21T11:36:58.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:256: TEST_ok_to_upgrade_bad_osd_version: local dir=td/ok-to-upgrade 2026-02-21T11:36:58.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:257: TEST_ok_to_upgrade_bad_osd_version: local poolname=test 2026-02-21T11:36:58.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:258: TEST_ok_to_upgrade_bad_osd_version: local OSDS=3 2026-02-21T11:36:58.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:259: TEST_ok_to_upgrade_bad_osd_version: local ceph_version=01.2.3-1234-g1234deed 2026-02-21T11:36:58.836 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:261: TEST_ok_to_upgrade_bad_osd_version: CEPH_ARGS='--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:36:58.836 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:263: TEST_ok_to_upgrade_bad_osd_version: run_mon td/ok-to-upgrade a --public-addr=127.0.0.1:7170 2026-02-21T11:36:58.836 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-upgrade 2026-02-21T11:36:58.836 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:36:58.836 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:36:58.836 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:36:58.836 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-upgrade/a 2026-02-21T11:36:58.836 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-upgrade/a --run-dir=td/ok-to-upgrade --public-addr=127.0.0.1:7170 2026-02-21T11:36:58.852 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:36:58.852 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:36:58.852 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:36:58.852 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:36:58.853 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:36:58.853 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:36:58.853 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:36:58.853 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-upgrade/a '--log-file=td/ok-to-upgrade/$name.log' '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-upgrade/log --run-dir=td/ok-to-upgrade '--pid-file=td/ok-to-upgrade/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7170 2026-02-21T11:36:58.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:36:58.878 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:36:58.878 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:36:58.878 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:36:58.878 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:36:58.878 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:36:58.879 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:36:58.879 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:36:58.879 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:36:58.879 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:36:58.879 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:36:58.879 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:36:58.879 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73624/ceph-mon.a.asok 2026-02-21T11:36:58.879 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:36:58.879 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73624/ceph-mon.a.asok config get fsid 2026-02-21T11:36:58.924 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:36:58.924 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:36:58.924 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:36:58.925 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:36:58.925 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:36:58.925 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:36:58.925 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:36:58.925 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:36:58.925 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:36:58.925 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:36:58.925 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:36:58.925 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73624/ceph-mon.a.asok 2026-02-21T11:36:58.925 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:36:58.925 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73624/ceph-mon.a.asok config get mon_host 2026-02-21T11:36:58.966 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:264: TEST_ok_to_upgrade_bad_osd_version: run_mgr td/ok-to-upgrade x 2026-02-21T11:36:58.966 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-upgrade 2026-02-21T11:36:58.966 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:36:58.966 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:36:58.966 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:36:58.966 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-upgrade/x 2026-02-21T11:36:58.966 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:36:59.065 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:36:59.065 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:36:59.065 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:36:59.065 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:36:59.065 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:36:59.066 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:36:59.066 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:36:59.066 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:36:59.066 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-upgrade/x '--log-file=td/ok-to-upgrade/$name.log' '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --run-dir=td/ok-to-upgrade '--pid-file=td/ok-to-upgrade/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:36:59.080 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:266: TEST_ok_to_upgrade_bad_osd_version: expr 3 - 1 2026-02-21T11:36:59.081 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:266: TEST_ok_to_upgrade_bad_osd_version: seq 0 2 2026-02-21T11:36:59.085 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:266: TEST_ok_to_upgrade_bad_osd_version: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:36:59.085 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:268: TEST_ok_to_upgrade_bad_osd_version: run_osd td/ok-to-upgrade 0 --osd-mclock-skip-benchmark=true 2026-02-21T11:36:59.085 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:36:59.085 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:36:59.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:36:59.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:36:59.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/0 2026-02-21T11:36:59.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:36:59.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:36:59.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:36:59.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:36:59.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/0' 2026-02-21T11:36:59.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/0/journal' 2026-02-21T11:36:59.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:36:59.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:36:59.087 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:36:59.087 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:36:59.087 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:36:59.087 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:36:59.087 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:36:59.087 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:36:59.087 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:36:59.087 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:36:59.087 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:36:59.087 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:36:59.087 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:36:59.087 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:36:59.088 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:36:59.088 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:36:59.088 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:36:59.088 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:36:59.088 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:36:59.088 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:36:59.088 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:36:59.088 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:36:59.088 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/0 2026-02-21T11:36:59.088 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:36:59.088 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6fbc5c46-4bd0-4de5-aadf-d022249321b8 2026-02-21T11:36:59.088 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 6fbc5c46-4bd0-4de5-aadf-d022249321b8' 2026-02-21T11:36:59.089 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 6fbc5c46-4bd0-4de5-aadf-d022249321b8 2026-02-21T11:36:59.089 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:36:59.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDbmJlpSk2JBRAAX5LpAWY5wpzqariMF0tu1Q== 2026-02-21T11:36:59.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDbmJlpSk2JBRAAX5LpAWY5wpzqariMF0tu1Q=="}' 2026-02-21T11:36:59.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6fbc5c46-4bd0-4de5-aadf-d022249321b8 -i td/ok-to-upgrade/0/new.json 2026-02-21T11:36:59.194 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:36:59.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/0/new.json 2026-02-21T11:36:59.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/0 --osd-journal=td/ok-to-upgrade/0/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQDbmJlpSk2JBRAAX5LpAWY5wpzqariMF0tu1Q== --osd-uuid 6fbc5c46-4bd0-4de5-aadf-d022249321b8 2026-02-21T11:36:59.216 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:36:59.214+0000 7f3d5e237780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:36:59.217 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:36:59.215+0000 7f3d5e237780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:36:59.218 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:36:59.216+0000 7f3d5e237780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:36:59.218 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:36:59.216+0000 7f3d5e237780 -1 bdev(0x33ccc00 td/ok-to-upgrade/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:36:59.218 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:36:59.216+0000 7f3d5e237780 -1 bluestore(td/ok-to-upgrade/0) _read_fsid unparsable uuid 2026-02-21T11:36:59.677 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/0/keyring 2026-02-21T11:36:59.677 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:36:59.678 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:36:59.678 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:36:59.679 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:36:59.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:36:59.783 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:36:59.784 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/0 --osd-journal=td/ok-to-upgrade/0/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:36:59.784 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:36:59.785 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:36:59.785 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:36:59.796 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:36:59.794+0000 7fbc70c4b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:36:59.798 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:36:59.796+0000 7fbc70c4b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:36:59.799 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:36:59.797+0000 7fbc70c4b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:36:59.880 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:36:59.880 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:36:59.880 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:36:59.881 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:36:59.881 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:36:59.881 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:36:59.881 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:36:59.882 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:36:59.882 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:36:59.883 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:36:59.976 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:36:59.979 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:36:59.977+0000 7fbc70c4b780 -1 Falling back to public interface 2026-02-21T11:37:00.369 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:00.367+0000 7fbc70c4b780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:37:00.977 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:37:00.977 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:00.977 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:37:00.977 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:37:00.978 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:00.978 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:37:01.074 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2795172975,v1:127.0.0.1:6801/2795172975] [v2:127.0.0.1:6802/2795172975,v1:127.0.0.1:6803/2795172975] exists,up 6fbc5c46-4bd0-4de5-aadf-d022249321b8 2026-02-21T11:37:01.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:37:01.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:37:01.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:37:01.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:266: TEST_ok_to_upgrade_bad_osd_version: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:37:01.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:268: TEST_ok_to_upgrade_bad_osd_version: run_osd td/ok-to-upgrade 1 --osd-mclock-skip-benchmark=true 2026-02-21T11:37:01.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:37:01.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:37:01.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T11:37:01.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:37:01.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/1 2026-02-21T11:37:01.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:37:01.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:37:01.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:37:01.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:37:01.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/1' 2026-02-21T11:37:01.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/1/journal' 2026-02-21T11:37:01.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:37:01.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:37:01.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:37:01.080 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:37:01.081 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:37:01.081 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:37:01.081 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:37:01.081 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:01.081 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:01.081 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:01.081 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:01.081 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:37:01.081 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:37:01.081 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:37:01.081 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:37:01.081 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:37:01.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:37:01.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:37:01.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:37:01.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:37:01.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:37:01.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:37:01.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/1 2026-02-21T11:37:01.082 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:37:01.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ecf2ac21-965a-4676-8fea-6b339a85c341 2026-02-21T11:37:01.083 INFO:tasks.workunit.client.0.trial189.stdout:add osd1 ecf2ac21-965a-4676-8fea-6b339a85c341 2026-02-21T11:37:01.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 ecf2ac21-965a-4676-8fea-6b339a85c341' 2026-02-21T11:37:01.083 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:37:01.087 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDdmJlpgNIXBRAA4TW2wsvEW2LXtg4UAXblGQ== 2026-02-21T11:37:01.087 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDdmJlpgNIXBRAA4TW2wsvEW2LXtg4UAXblGQ=="}' 2026-02-21T11:37:01.087 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ecf2ac21-965a-4676-8fea-6b339a85c341 -i td/ok-to-upgrade/1/new.json 2026-02-21T11:37:01.189 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:37:01.194 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/1/new.json 2026-02-21T11:37:01.195 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/1 --osd-journal=td/ok-to-upgrade/1/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQDdmJlpgNIXBRAA4TW2wsvEW2LXtg4UAXblGQ== --osd-uuid ecf2ac21-965a-4676-8fea-6b339a85c341 2026-02-21T11:37:01.211 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:01.210+0000 7fcc9380e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:01.213 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:01.211+0000 7fcc9380e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:01.213 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:01.212+0000 7fcc9380e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:01.214 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:01.212+0000 7fcc9380e780 -1 bdev(0x41fcc00 td/ok-to-upgrade/1/block) open stat got: (1) Operation not permitted 2026-02-21T11:37:01.214 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:01.212+0000 7fcc9380e780 -1 bluestore(td/ok-to-upgrade/1) _read_fsid unparsable uuid 2026-02-21T11:37:01.539 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/1/keyring 2026-02-21T11:37:01.540 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:37:01.540 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T11:37:01.540 INFO:tasks.workunit.client.0.trial189.stdout:adding osd1 key to auth repository 2026-02-21T11:37:01.541 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:37:01.855 INFO:tasks.workunit.client.0.trial189.stdout:start osd.1 2026-02-21T11:37:01.856 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T11:37:01.856 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/1 --osd-journal=td/ok-to-upgrade/1/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:37:01.856 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:37:01.856 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:37:01.856 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:37:01.868 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:01.866+0000 7f646423a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:01.869 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:01.867+0000 7f646423a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:01.870 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:01.868+0000 7f646423a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:01.997 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:01.995+0000 7f646423a780 -1 Falling back to public interface 2026-02-21T11:37:02.078 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T11:37:02.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:37:02.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T11:37:02.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:37:02.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:37:02.079 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:37:02.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:02.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:37:02.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:02.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:37:02.302 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:02.300+0000 7f646423a780 -1 osd.1 0 log_to_monitors true 2026-02-21T11:37:02.304 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:37:03.306 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:37:03.306 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:03.306 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:37:03.306 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:37:03.306 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:03.306 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:37:03.532 INFO:tasks.workunit.client.0.trial189.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2494957666,v1:127.0.0.1:6811/2494957666] [v2:127.0.0.1:6812/2494957666,v1:127.0.0.1:6813/2494957666] exists,up ecf2ac21-965a-4676-8fea-6b339a85c341 2026-02-21T11:37:03.535 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:37:03.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:37:03.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:37:03.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:266: TEST_ok_to_upgrade_bad_osd_version: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:37:03.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:268: TEST_ok_to_upgrade_bad_osd_version: run_osd td/ok-to-upgrade 2 --osd-mclock-skip-benchmark=true 2026-02-21T11:37:03.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:37:03.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:37:03.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T11:37:03.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:37:03.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/2 2026-02-21T11:37:03.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:37:03.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/2' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/2/journal' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/2 2026-02-21T11:37:03.537 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:37:03.538 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=51d2e83c-a526-4789-98d3-f30e665ed6c0 2026-02-21T11:37:03.538 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 51d2e83c-a526-4789-98d3-f30e665ed6c0' 2026-02-21T11:37:03.538 INFO:tasks.workunit.client.0.trial189.stdout:add osd2 51d2e83c-a526-4789-98d3-f30e665ed6c0 2026-02-21T11:37:03.538 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:37:03.545 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDfmJlpkNFvIBAANOH6Mf8FtqhnPfxF2WKpVw== 2026-02-21T11:37:03.545 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDfmJlpkNFvIBAANOH6Mf8FtqhnPfxF2WKpVw=="}' 2026-02-21T11:37:03.545 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 51d2e83c-a526-4789-98d3-f30e665ed6c0 -i td/ok-to-upgrade/2/new.json 2026-02-21T11:37:03.774 INFO:tasks.workunit.client.0.trial189.stdout:2 2026-02-21T11:37:03.781 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/2/new.json 2026-02-21T11:37:03.782 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/2 --osd-journal=td/ok-to-upgrade/2/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQDfmJlpkNFvIBAANOH6Mf8FtqhnPfxF2WKpVw== --osd-uuid 51d2e83c-a526-4789-98d3-f30e665ed6c0 2026-02-21T11:37:03.797 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:03.795+0000 7feb32442780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:03.798 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:03.796+0000 7feb32442780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:03.799 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:03.797+0000 7feb32442780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:03.799 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:03.797+0000 7feb32442780 -1 bdev(0x31e0c00 td/ok-to-upgrade/2/block) open stat got: (1) Operation not permitted 2026-02-21T11:37:03.799 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:03.797+0000 7feb32442780 -1 bluestore(td/ok-to-upgrade/2) _read_fsid unparsable uuid 2026-02-21T11:37:04.119 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/2/keyring 2026-02-21T11:37:04.119 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:37:04.119 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T11:37:04.119 INFO:tasks.workunit.client.0.trial189.stdout:adding osd2 key to auth repository 2026-02-21T11:37:04.120 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:37:04.440 INFO:tasks.workunit.client.0.trial189.stdout:start osd.2 2026-02-21T11:37:04.441 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T11:37:04.441 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/2 --osd-journal=td/ok-to-upgrade/2/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:37:04.441 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:37:04.442 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:37:04.442 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:37:04.454 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:04.453+0000 7f0704a0c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:04.456 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:04.454+0000 7f0704a0c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:04.457 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:04.455+0000 7f0704a0c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:04.594 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:04.592+0000 7f0704a0c780 -1 Falling back to public interface 2026-02-21T11:37:04.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T11:37:04.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:37:04.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T11:37:04.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:37:04.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:37:04.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:04.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:37:04.665 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:37:04.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:04.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T11:37:04.883 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:37:04.932 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:04.930+0000 7f0704a0c780 -1 osd.2 0 log_to_monitors true 2026-02-21T11:37:05.885 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:37:05.885 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:05.885 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:37:05.885 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:37:05.885 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:05.885 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T11:37:06.111 INFO:tasks.workunit.client.0.trial189.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/272608564,v1:127.0.0.1:6819/272608564] [v2:127.0.0.1:6820/272608564,v1:127.0.0.1:6821/272608564] exists,up 51d2e83c-a526-4789-98d3-f30e665ed6c0 2026-02-21T11:37:06.111 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:37:06.111 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:37:06.111 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:37:06.111 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:271: TEST_ok_to_upgrade_bad_osd_version: create_pool test 8 8 2026-02-21T11:37:06.111 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 8 8 2026-02-21T11:37:06.356 INFO:tasks.workunit.client.0.trial189.stderr:pool 'test' already exists 2026-02-21T11:37:06.363 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:37:07.365 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:272: TEST_ok_to_upgrade_bad_osd_version: ceph osd pool set test min_size 1 2026-02-21T11:37:07.680 INFO:tasks.workunit.client.0.trial189.stderr:set pool 1 min_size to 1 2026-02-21T11:37:07.692 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:273: TEST_ok_to_upgrade_bad_osd_version: sleep 5 2026-02-21T11:37:12.694 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:275: TEST_ok_to_upgrade_bad_osd_version: wait_for_clean 2026-02-21T11:37:12.694 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:37:12.695 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:37:12.695 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:37:12.695 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:37:12.696 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:37:12.696 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:37:12.696 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:37:12.696 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:37:12.696 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:37:12.779 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:37:12.779 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:37:12.779 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:37:12.779 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:37:12.779 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:37:12.779 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:37:12.999 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:37:13.000 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:37:13.000 INFO:tasks.workunit.client.0.trial189.stderr:2' 2026-02-21T11:37:13.000 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:37:13.000 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:13.000 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:37:13.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836485 2026-02-21T11:37:13.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836485 2026-02-21T11:37:13.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836485' 2026-02-21T11:37:13.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:13.061 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:37:13.123 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=38654705668 2026-02-21T11:37:13.123 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 38654705668 2026-02-21T11:37:13.123 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836485 1-38654705668' 2026-02-21T11:37:13.123 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:13.124 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-21T11:37:13.184 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=55834574852 2026-02-21T11:37:13.184 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 55834574852 2026-02-21T11:37:13.184 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836485 1-38654705668 2-55834574852' 2026-02-21T11:37:13.184 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:37:13.184 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836485 2026-02-21T11:37:13.184 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:37:13.185 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:37:13.186 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836485 2026-02-21T11:37:13.186 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:37:13.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836485 2026-02-21T11:37:13.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-21T11:37:13.187 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836485 2026-02-21T11:37:13.187 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:37:13.412 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-21T11:37:13.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:37:13.413 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-38654705668 2026-02-21T11:37:13.413 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:37:13.414 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:37:13.414 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-38654705668 2026-02-21T11:37:13.415 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:37:13.416 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=38654705668 2026-02-21T11:37:13.416 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 38654705668' 2026-02-21T11:37:13.416 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 38654705668 2026-02-21T11:37:13.416 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:37:13.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 38654705666 -lt 38654705668 2026-02-21T11:37:13.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:37:14.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:37:14.645 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:37:14.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 38654705666 -lt 38654705668 2026-02-21T11:37:14.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:37:15.871 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T11:37:15.872 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:37:16.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 38654705669 -lt 38654705668 2026-02-21T11:37:16.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:37:16.095 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 2-55834574852 2026-02-21T11:37:16.095 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:37:16.097 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=2 2026-02-21T11:37:16.097 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 2-55834574852 2026-02-21T11:37:16.097 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:37:16.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=55834574852 2026-02-21T11:37:16.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.2 seq 55834574852' 2026-02-21T11:37:16.098 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.2 seq 55834574852 2026-02-21T11:37:16.098 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T11:37:16.321 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 55834574852 -lt 55834574852 2026-02-21T11:37:16.321 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:37:16.321 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:37:16.322 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:37:16.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 8 == 0 2026-02-21T11:37:16.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:37:16.636 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:37:16.636 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:37:16.636 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:37:16.636 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:37:16.636 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:37:16.636 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:37:16.855 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=8 2026-02-21T11:37:16.856 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:37:16.856 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:37:16.856 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:37:17.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 8 = 8 2026-02-21T11:37:17.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:37:17.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:37:17.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:278: TEST_ok_to_upgrade_bad_osd_version: ceph config set mgr mgr_test_metadata_error true 2026-02-21T11:37:17.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:281: TEST_ok_to_upgrade_bad_osd_version: local exp_osds_upgradable=0 2026-02-21T11:37:17.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:282: TEST_ok_to_upgrade_bad_osd_version: local exp_osds_bad_version=3 2026-02-21T11:37:17.395 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:283: TEST_ok_to_upgrade_bad_osd_version: ceph osd tree 2026-02-21T11:37:17.395 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:283: TEST_ok_to_upgrade_bad_osd_version: grep host 2026-02-21T11:37:17.395 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:283: TEST_ok_to_upgrade_bad_osd_version: awk '{ print $4 }' 2026-02-21T11:37:17.619 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:283: TEST_ok_to_upgrade_bad_osd_version: local crush_bucket=trial189 2026-02-21T11:37:17.619 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:284: TEST_ok_to_upgrade_bad_osd_version: ceph osd ok-to-upgrade trial189 01.2.3-1234-g1234deed --format=json 2026-02-21T11:37:17.778 INFO:tasks.workunit.client.0.trial189.stderr:Error EAGAIN: 3 osds have unknown version; cannot draw any conclusions 2026-02-21T11:37:17.781 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:284: TEST_ok_to_upgrade_bad_osd_version: local 'res= 2026-02-21T11:37:17.781 INFO:tasks.workunit.client.0.trial189.stderr:{"ok_to_upgrade":false,"all_osds_upgraded":false,"osds_in_crush_bucket":[0,1,2],"osds_ok_to_upgrade":[],"osds_upgraded":[],"bad_no_version":[0,1,2]}' 2026-02-21T11:37:17.782 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:285: TEST_ok_to_upgrade_bad_osd_version: echo '{"ok_to_upgrade":false,"all_osds_upgraded":false,"osds_in_crush_bucket":[0,1,2],"osds_ok_to_upgrade":[],"osds_upgraded":[],"bad_no_version":[0,1,2]}' 2026-02-21T11:37:17.782 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:285: TEST_ok_to_upgrade_bad_osd_version: jq .all_osds_upgraded 2026-02-21T11:37:17.784 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:285: TEST_ok_to_upgrade_bad_osd_version: test false = false 2026-02-21T11:37:17.785 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:286: TEST_ok_to_upgrade_bad_osd_version: echo '{"ok_to_upgrade":false,"all_osds_upgraded":false,"osds_in_crush_bucket":[0,1,2],"osds_ok_to_upgrade":[],"osds_upgraded":[],"bad_no_version":[0,1,2]}' 2026-02-21T11:37:17.785 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:286: TEST_ok_to_upgrade_bad_osd_version: jq .ok_to_upgrade 2026-02-21T11:37:17.787 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:286: TEST_ok_to_upgrade_bad_osd_version: test false = false 2026-02-21T11:37:17.788 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:287: TEST_ok_to_upgrade_bad_osd_version: echo '{"ok_to_upgrade":false,"all_osds_upgraded":false,"osds_in_crush_bucket":[0,1,2],"osds_ok_to_upgrade":[],"osds_upgraded":[],"bad_no_version":[0,1,2]}' 2026-02-21T11:37:17.788 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:287: TEST_ok_to_upgrade_bad_osd_version: jq '.osds_ok_to_upgrade | length' 2026-02-21T11:37:17.788 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:287: TEST_ok_to_upgrade_bad_osd_version: bc 2026-02-21T11:37:17.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:287: TEST_ok_to_upgrade_bad_osd_version: local num_osds_upgradable=0 2026-02-21T11:37:17.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:288: TEST_ok_to_upgrade_bad_osd_version: test 0 = 0 2026-02-21T11:37:17.791 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:289: TEST_ok_to_upgrade_bad_osd_version: echo '{"ok_to_upgrade":false,"all_osds_upgraded":false,"osds_in_crush_bucket":[0,1,2],"osds_ok_to_upgrade":[],"osds_upgraded":[],"bad_no_version":[0,1,2]}' 2026-02-21T11:37:17.791 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:289: TEST_ok_to_upgrade_bad_osd_version: jq '.bad_no_version | length' 2026-02-21T11:37:17.791 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:289: TEST_ok_to_upgrade_bad_osd_version: bc 2026-02-21T11:37:17.793 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:289: TEST_ok_to_upgrade_bad_osd_version: local num_osds_bad_version=3 2026-02-21T11:37:17.793 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:290: TEST_ok_to_upgrade_bad_osd_version: test 3 -eq 3 2026-02-21T11:37:17.793 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:33: run: kill_daemons td/ok-to-upgrade KILL 2026-02-21T11:37:17.794 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:37:17.794 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:37:17.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:37:17.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:37:17.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:37:17.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:37:17.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:34: run: teardown td/ok-to-upgrade 2026-02-21T11:37:17.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-upgrade 2026-02-21T11:37:17.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:37:17.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-upgrade KILL 2026-02-21T11:37:17.901 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:37:17.901 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:37:17.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:37:17.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:37:17.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:37:17.905 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:37:17.905 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:37:17.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:37:17.906 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:37:17.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:37:17.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:37:17.907 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:37:17.908 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:37:17.908 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:37:17.908 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:37:17.908 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:37:17.909 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:37:17.910 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:37:17.910 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-upgrade 2026-02-21T11:37:17.916 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:37:17.916 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:17.916 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:17.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73624 2026-02-21T11:37:17.917 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:37:17.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:37:17.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:30: run: for func in $funcs 2026-02-21T11:37:17.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:31: run: setup td/ok-to-upgrade 2026-02-21T11:37:17.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-upgrade 2026-02-21T11:37:17.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-upgrade 2026-02-21T11:37:17.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-upgrade 2026-02-21T11:37:17.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:37:17.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-upgrade KILL 2026-02-21T11:37:17.918 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:37:17.918 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:37:17.919 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:37:17.919 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:37:17.919 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:37:17.920 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:37:17.920 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:37:17.920 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:37:17.921 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:37:17.922 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:37:17.922 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:37:17.922 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:37:17.923 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:37:17.923 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:37:17.923 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:37:17.924 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:37:17.924 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:37:17.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:37:17.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-upgrade 2026-02-21T11:37:17.926 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:37:17.926 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:17.926 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:17.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73624 2026-02-21T11:37:17.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:37:17.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:37:17.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-upgrade 2026-02-21T11:37:17.928 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:37:17.928 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:17.928 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:17.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73624 2026-02-21T11:37:17.929 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:37:17.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:37:17.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:37:17.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-upgrade 1' TERM HUP INT 2026-02-21T11:37:17.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:32: run: TEST_ok_to_upgrade_erasure_pool td/ok-to-upgrade 2026-02-21T11:37:17.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:164: TEST_ok_to_upgrade_erasure_pool: local dir=td/ok-to-upgrade 2026-02-21T11:37:17.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:165: TEST_ok_to_upgrade_erasure_pool: local poolname=ec 2026-02-21T11:37:17.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:166: TEST_ok_to_upgrade_erasure_pool: local OSDS=10 2026-02-21T11:37:17.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:167: TEST_ok_to_upgrade_erasure_pool: local ceph_version=01.2.3-1234-g1234deed 2026-02-21T11:37:17.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:169: TEST_ok_to_upgrade_erasure_pool: CEPH_ARGS='--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:37:17.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:171: TEST_ok_to_upgrade_erasure_pool: run_mon td/ok-to-upgrade a --public-addr=127.0.0.1:7170 2026-02-21T11:37:17.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-upgrade 2026-02-21T11:37:17.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:37:17.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:37:17.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:37:17.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-upgrade/a 2026-02-21T11:37:17.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-upgrade/a --run-dir=td/ok-to-upgrade --public-addr=127.0.0.1:7170 2026-02-21T11:37:17.948 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:37:17.949 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:37:17.949 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:37:17.949 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:37:17.949 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:17.949 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:17.949 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:17.949 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-upgrade/a '--log-file=td/ok-to-upgrade/$name.log' '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-upgrade/log --run-dir=td/ok-to-upgrade '--pid-file=td/ok-to-upgrade/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7170 2026-02-21T11:37:17.972 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:37:17.973 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:37:17.973 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:37:17.974 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:37:17.974 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:37:17.974 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:37:17.974 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:37:17.974 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:37:17.974 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:37:17.975 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:37:17.975 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:17.975 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:17.975 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73624/ceph-mon.a.asok 2026-02-21T11:37:17.975 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:37:17.976 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73624/ceph-mon.a.asok config get fsid 2026-02-21T11:37:18.018 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:37:18.018 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:37:18.018 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:37:18.018 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:37:18.018 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:37:18.018 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:37:18.018 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:37:18.018 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:37:18.018 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:37:18.019 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:18.019 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:18.019 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73624/ceph-mon.a.asok 2026-02-21T11:37:18.019 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:37:18.019 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73624/ceph-mon.a.asok config get mon_host 2026-02-21T11:37:18.059 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:172: TEST_ok_to_upgrade_erasure_pool: run_mgr td/ok-to-upgrade x 2026-02-21T11:37:18.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-upgrade 2026-02-21T11:37:18.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:37:18.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:37:18.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:37:18.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-upgrade/x 2026-02-21T11:37:18.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:37:18.153 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:37:18.153 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:37:18.153 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:37:18.154 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:37:18.154 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:18.154 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:18.154 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:18.154 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:37:18.155 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-upgrade/x '--log-file=td/ok-to-upgrade/$name.log' '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --run-dir=td/ok-to-upgrade '--pid-file=td/ok-to-upgrade/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:37:18.171 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:174: TEST_ok_to_upgrade_erasure_pool: expr 10 - 1 2026-02-21T11:37:18.171 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:174: TEST_ok_to_upgrade_erasure_pool: seq 0 9 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:174: TEST_ok_to_upgrade_erasure_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:176: TEST_ok_to_upgrade_erasure_pool: run_osd td/ok-to-upgrade 0 --osd-mclock-skip-benchmark=true 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/0 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/0' 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/0/journal' 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:37:18.175 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/0 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6271dfc7-8082-421b-90d4-5c254b402d07 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 6271dfc7-8082-421b-90d4-5c254b402d07 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 6271dfc7-8082-421b-90d4-5c254b402d07' 2026-02-21T11:37:18.176 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:37:18.185 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDumJlpIi30ChAAYPuCECDlPAt6ft25fLD4vQ== 2026-02-21T11:37:18.185 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDumJlpIi30ChAAYPuCECDlPAt6ft25fLD4vQ=="}' 2026-02-21T11:37:18.185 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6271dfc7-8082-421b-90d4-5c254b402d07 -i td/ok-to-upgrade/0/new.json 2026-02-21T11:37:18.283 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:37:18.289 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/0/new.json 2026-02-21T11:37:18.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/0 --osd-journal=td/ok-to-upgrade/0/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQDumJlpIi30ChAAYPuCECDlPAt6ft25fLD4vQ== --osd-uuid 6271dfc7-8082-421b-90d4-5c254b402d07 2026-02-21T11:37:18.305 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:18.303+0000 7f830ea17780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:18.307 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:18.305+0000 7f830ea17780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:18.307 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:18.306+0000 7f830ea17780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:18.308 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:18.306+0000 7f830ea17780 -1 bdev(0x34f8c00 td/ok-to-upgrade/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:37:18.308 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:18.306+0000 7f830ea17780 -1 bluestore(td/ok-to-upgrade/0) _read_fsid unparsable uuid 2026-02-21T11:37:18.755 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/0/keyring 2026-02-21T11:37:18.755 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:37:18.756 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:37:18.756 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:37:18.756 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:37:18.858 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:37:18.859 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:37:18.859 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/0 --osd-journal=td/ok-to-upgrade/0/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:37:18.859 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:37:18.859 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:37:18.860 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:37:18.872 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:18.870+0000 7fe470c3a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:18.873 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:18.872+0000 7fe470c3a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:18.874 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:18.873+0000 7fe470c3a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:18.954 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:37:18.954 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:37:18.954 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:37:18.954 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:37:18.954 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:37:18.954 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:18.954 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:37:18.954 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:37:18.955 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:18.955 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:37:19.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:37:19.061 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:19.060+0000 7fe470c3a780 -1 Falling back to public interface 2026-02-21T11:37:19.414 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:19.412+0000 7fe470c3a780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:37:20.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:37:20.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:20.052 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:37:20.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:37:20.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:20.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:37:20.147 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3761227088,v1:127.0.0.1:6801/3761227088] [v2:127.0.0.1:6802/3761227088,v1:127.0.0.1:6803/3761227088] exists,up 6271dfc7-8082-421b-90d4-5c254b402d07 2026-02-21T11:37:20.151 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:37:20.151 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:37:20.151 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:37:20.151 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:174: TEST_ok_to_upgrade_erasure_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:37:20.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:176: TEST_ok_to_upgrade_erasure_pool: run_osd td/ok-to-upgrade 1 --osd-mclock-skip-benchmark=true 2026-02-21T11:37:20.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:37:20.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:37:20.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T11:37:20.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:37:20.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/1 2026-02-21T11:37:20.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:37:20.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:37:20.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:37:20.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:37:20.152 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/1' 2026-02-21T11:37:20.153 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/1/journal' 2026-02-21T11:37:20.153 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:37:20.153 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:37:20.153 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:37:20.153 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:37:20.153 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:37:20.153 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:37:20.153 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:37:20.153 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:20.153 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:20.153 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:20.153 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:20.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:37:20.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:37:20.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:37:20.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:37:20.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:37:20.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:37:20.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:37:20.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:37:20.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:37:20.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:37:20.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:37:20.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/1 2026-02-21T11:37:20.154 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:37:20.155 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e6b471c8-9a8a-4fdb-b73b-f308953e4928 2026-02-21T11:37:20.155 INFO:tasks.workunit.client.0.trial189.stdout:add osd1 e6b471c8-9a8a-4fdb-b73b-f308953e4928 2026-02-21T11:37:20.156 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 e6b471c8-9a8a-4fdb-b73b-f308953e4928' 2026-02-21T11:37:20.156 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:37:20.160 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDwmJlp0gF6CRAA48Shc4pLpE2f2YrQDeyTWA== 2026-02-21T11:37:20.160 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDwmJlp0gF6CRAA48Shc4pLpE2f2YrQDeyTWA=="}' 2026-02-21T11:37:20.160 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e6b471c8-9a8a-4fdb-b73b-f308953e4928 -i td/ok-to-upgrade/1/new.json 2026-02-21T11:37:20.261 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:37:20.266 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/1/new.json 2026-02-21T11:37:20.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/1 --osd-journal=td/ok-to-upgrade/1/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQDwmJlp0gF6CRAA48Shc4pLpE2f2YrQDeyTWA== --osd-uuid e6b471c8-9a8a-4fdb-b73b-f308953e4928 2026-02-21T11:37:20.281 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:20.280+0000 7fb899210780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:20.283 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:20.281+0000 7fb899210780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:20.284 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:20.282+0000 7fb899210780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:20.284 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:20.282+0000 7fb899210780 -1 bdev(0x4368c00 td/ok-to-upgrade/1/block) open stat got: (1) Operation not permitted 2026-02-21T11:37:20.284 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:20.282+0000 7fb899210780 -1 bluestore(td/ok-to-upgrade/1) _read_fsid unparsable uuid 2026-02-21T11:37:20.595 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/1/keyring 2026-02-21T11:37:20.595 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:37:20.596 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T11:37:20.596 INFO:tasks.workunit.client.0.trial189.stdout:adding osd1 key to auth repository 2026-02-21T11:37:20.596 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:37:20.912 INFO:tasks.workunit.client.0.trial189.stdout:start osd.1 2026-02-21T11:37:20.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T11:37:20.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/1 --osd-journal=td/ok-to-upgrade/1/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:37:20.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:37:20.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:37:20.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:37:20.926 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:20.924+0000 7fc4a8c60780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:20.927 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:20.926+0000 7fc4a8c60780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:20.928 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:20.927+0000 7fc4a8c60780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:21.049 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:21.047+0000 7fc4a8c60780 -1 Falling back to public interface 2026-02-21T11:37:21.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T11:37:21.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:37:21.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T11:37:21.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:37:21.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:37:21.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:21.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:37:21.137 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:37:21.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:21.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:37:21.358 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:37:21.371 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:21.370+0000 7fc4a8c60780 -1 osd.1 0 log_to_monitors true 2026-02-21T11:37:22.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:37:22.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:22.359 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:37:22.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:37:22.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:22.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:37:22.587 INFO:tasks.workunit.client.0.trial189.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1079757845,v1:127.0.0.1:6811/1079757845] [v2:127.0.0.1:6812/1079757845,v1:127.0.0.1:6813/1079757845] exists,up e6b471c8-9a8a-4fdb-b73b-f308953e4928 2026-02-21T11:37:22.591 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:37:22.591 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:37:22.591 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:37:22.591 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:174: TEST_ok_to_upgrade_erasure_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:37:22.591 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:176: TEST_ok_to_upgrade_erasure_pool: run_osd td/ok-to-upgrade 2 --osd-mclock-skip-benchmark=true 2026-02-21T11:37:22.591 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:37:22.591 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:37:22.591 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T11:37:22.591 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:37:22.591 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/2 2026-02-21T11:37:22.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:37:22.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:37:22.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:37:22.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:37:22.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/2' 2026-02-21T11:37:22.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/2/journal' 2026-02-21T11:37:22.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:37:22.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:37:22.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:37:22.592 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:37:22.592 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:37:22.593 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:37:22.593 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:37:22.593 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:22.593 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:22.593 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:22.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:22.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:37:22.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:37:22.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:37:22.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:37:22.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:37:22.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:37:22.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:37:22.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:37:22.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:37:22.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:37:22.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:37:22.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/2 2026-02-21T11:37:22.594 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:37:22.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6a2283b2-0693-4a15-89b1-9d1ec6a92a66 2026-02-21T11:37:22.594 INFO:tasks.workunit.client.0.trial189.stdout:add osd2 6a2283b2-0693-4a15-89b1-9d1ec6a92a66 2026-02-21T11:37:22.595 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 6a2283b2-0693-4a15-89b1-9d1ec6a92a66' 2026-02-21T11:37:22.595 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:37:22.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDymJlpIl6qIxAAgX7HoEF8tGfoXRUzSuHP+w== 2026-02-21T11:37:22.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDymJlpIl6qIxAAgX7HoEF8tGfoXRUzSuHP+w=="}' 2026-02-21T11:37:22.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6a2283b2-0693-4a15-89b1-9d1ec6a92a66 -i td/ok-to-upgrade/2/new.json 2026-02-21T11:37:22.827 INFO:tasks.workunit.client.0.trial189.stdout:2 2026-02-21T11:37:22.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/2/new.json 2026-02-21T11:37:22.836 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/2 --osd-journal=td/ok-to-upgrade/2/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQDymJlpIl6qIxAAgX7HoEF8tGfoXRUzSuHP+w== --osd-uuid 6a2283b2-0693-4a15-89b1-9d1ec6a92a66 2026-02-21T11:37:22.851 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:22.849+0000 7fb023c66780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:22.852 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:22.850+0000 7fb023c66780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:22.853 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:22.851+0000 7fb023c66780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:22.853 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:22.852+0000 7fb023c66780 -1 bdev(0x3d9ac00 td/ok-to-upgrade/2/block) open stat got: (1) Operation not permitted 2026-02-21T11:37:22.854 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:22.852+0000 7fb023c66780 -1 bluestore(td/ok-to-upgrade/2) _read_fsid unparsable uuid 2026-02-21T11:37:23.172 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/2/keyring 2026-02-21T11:37:23.172 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:37:23.173 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T11:37:23.173 INFO:tasks.workunit.client.0.trial189.stdout:adding osd2 key to auth repository 2026-02-21T11:37:23.173 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:37:23.491 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T11:37:23.492 INFO:tasks.workunit.client.0.trial189.stdout:start osd.2 2026-02-21T11:37:23.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/2 --osd-journal=td/ok-to-upgrade/2/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:37:23.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:37:23.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:37:23.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:37:23.505 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:23.503+0000 7fbcf4c36780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:23.506 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:23.505+0000 7fbcf4c36780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:23.507 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:23.506+0000 7fbcf4c36780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:23.657 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:23.655+0000 7fbcf4c36780 -1 Falling back to public interface 2026-02-21T11:37:23.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T11:37:23.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:37:23.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T11:37:23.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:37:23.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:37:23.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:23.715 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:37:23.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:37:23.716 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:23.716 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T11:37:23.938 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:37:24.025 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:24.023+0000 7fbcf4c36780 -1 osd.2 0 log_to_monitors true 2026-02-21T11:37:24.939 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:37:24.939 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:24.939 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:37:24.939 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:37:24.939 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:24.939 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T11:37:25.164 INFO:tasks.workunit.client.0.trial189.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2192241827,v1:127.0.0.1:6819/2192241827] [v2:127.0.0.1:6820/2192241827,v1:127.0.0.1:6821/2192241827] exists,up 6a2283b2-0693-4a15-89b1-9d1ec6a92a66 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:174: TEST_ok_to_upgrade_erasure_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:176: TEST_ok_to_upgrade_erasure_pool: run_osd td/ok-to-upgrade 3 --osd-mclock-skip-benchmark=true 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/3 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/3' 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/3/journal' 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:37:25.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:37:25.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:37:25.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:37:25.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:37:25.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:37:25.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:37:25.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:37:25.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:37:25.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:37:25.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:37:25.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/3 2026-02-21T11:37:25.169 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:37:25.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9845db9a-c460-4f92-81c5-0d0335178b9f 2026-02-21T11:37:25.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 9845db9a-c460-4f92-81c5-0d0335178b9f' 2026-02-21T11:37:25.169 INFO:tasks.workunit.client.0.trial189.stdout:add osd3 9845db9a-c460-4f92-81c5-0d0335178b9f 2026-02-21T11:37:25.169 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:37:25.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD1mJlpcSdvChAAobXnipOg8J4CfrXva9jqHQ== 2026-02-21T11:37:25.177 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD1mJlpcSdvChAAobXnipOg8J4CfrXva9jqHQ=="}' 2026-02-21T11:37:25.177 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9845db9a-c460-4f92-81c5-0d0335178b9f -i td/ok-to-upgrade/3/new.json 2026-02-21T11:37:25.399 INFO:tasks.workunit.client.0.trial189.stdout:3 2026-02-21T11:37:25.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/3/new.json 2026-02-21T11:37:25.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/3 --osd-journal=td/ok-to-upgrade/3/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQD1mJlpcSdvChAAobXnipOg8J4CfrXva9jqHQ== --osd-uuid 9845db9a-c460-4f92-81c5-0d0335178b9f 2026-02-21T11:37:25.421 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:25.419+0000 7f02d8246780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:25.422 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:25.420+0000 7f02d8246780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:25.423 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:25.421+0000 7f02d8246780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:25.423 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:25.422+0000 7f02d8246780 -1 bdev(0x3e84c00 td/ok-to-upgrade/3/block) open stat got: (1) Operation not permitted 2026-02-21T11:37:25.423 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:25.422+0000 7f02d8246780 -1 bluestore(td/ok-to-upgrade/3) _read_fsid unparsable uuid 2026-02-21T11:37:25.750 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/3/keyring 2026-02-21T11:37:25.750 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:37:25.751 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-21T11:37:25.751 INFO:tasks.workunit.client.0.trial189.stdout:adding osd3 key to auth repository 2026-02-21T11:37:25.751 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:37:26.068 INFO:tasks.workunit.client.0.trial189.stdout:start osd.3 2026-02-21T11:37:26.069 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-21T11:37:26.069 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/3 --osd-journal=td/ok-to-upgrade/3/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:37:26.069 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:37:26.069 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:37:26.069 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:37:26.081 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:26.079+0000 7fc425846780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:26.082 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:26.081+0000 7fc425846780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:26.083 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:26.082+0000 7fc425846780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:26.220 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:26.218+0000 7fc425846780 -1 Falling back to public interface 2026-02-21T11:37:26.292 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-21T11:37:26.292 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:37:26.292 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-21T11:37:26.292 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:37:26.293 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:37:26.293 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:26.293 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:37:26.294 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:37:26.294 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:26.294 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-21T11:37:26.516 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:37:26.579 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:26.578+0000 7fc425846780 -1 osd.3 0 log_to_monitors true 2026-02-21T11:37:27.516 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:37:27.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:27.517 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:37:27.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:37:27.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:27.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-21T11:37:27.740 INFO:tasks.workunit.client.0.trial189.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/559657720,v1:127.0.0.1:6827/559657720] [v2:127.0.0.1:6828/559657720,v1:127.0.0.1:6829/559657720] exists,up 9845db9a-c460-4f92-81c5-0d0335178b9f 2026-02-21T11:37:27.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:37:27.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:37:27.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:37:27.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:174: TEST_ok_to_upgrade_erasure_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:37:27.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:176: TEST_ok_to_upgrade_erasure_pool: run_osd td/ok-to-upgrade 4 --osd-mclock-skip-benchmark=true 2026-02-21T11:37:27.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:37:27.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:37:27.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-21T11:37:27.745 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:37:27.745 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/4 2026-02-21T11:37:27.745 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:37:27.745 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:37:27.745 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:37:27.745 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:37:27.745 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/4' 2026-02-21T11:37:27.745 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/4/journal' 2026-02-21T11:37:27.745 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:37:27.745 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:37:27.745 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:37:27.745 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:37:27.745 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:37:27.746 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:37:27.746 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:37:27.746 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:27.746 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:27.746 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:27.746 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:27.746 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:37:27.746 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:37:27.746 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:37:27.746 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:37:27.746 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:37:27.746 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:37:27.746 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:37:27.747 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:37:27.747 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:37:27.747 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:37:27.747 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:37:27.747 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/4 2026-02-21T11:37:27.747 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:37:27.747 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fc759a60-6b17-44cb-a535-c04c614c87fe 2026-02-21T11:37:27.747 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 fc759a60-6b17-44cb-a535-c04c614c87fe' 2026-02-21T11:37:27.747 INFO:tasks.workunit.client.0.trial189.stdout:add osd4 fc759a60-6b17-44cb-a535-c04c614c87fe 2026-02-21T11:37:27.748 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:37:27.752 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD3mJlpA1DGLBAAFRjAkKay/yVTbhfUFoGZcg== 2026-02-21T11:37:27.752 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD3mJlpA1DGLBAAFRjAkKay/yVTbhfUFoGZcg=="}' 2026-02-21T11:37:27.752 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fc759a60-6b17-44cb-a535-c04c614c87fe -i td/ok-to-upgrade/4/new.json 2026-02-21T11:37:27.978 INFO:tasks.workunit.client.0.trial189.stdout:4 2026-02-21T11:37:27.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/4/new.json 2026-02-21T11:37:27.987 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/4 --osd-journal=td/ok-to-upgrade/4/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQD3mJlpA1DGLBAAFRjAkKay/yVTbhfUFoGZcg== --osd-uuid fc759a60-6b17-44cb-a535-c04c614c87fe 2026-02-21T11:37:28.001 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:28.000+0000 7f9a06e3c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:28.003 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:28.001+0000 7f9a06e3c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:28.004 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:28.002+0000 7f9a06e3c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:28.004 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:28.002+0000 7f9a06e3c780 -1 bdev(0x334cc00 td/ok-to-upgrade/4/block) open stat got: (1) Operation not permitted 2026-02-21T11:37:28.004 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:28.002+0000 7f9a06e3c780 -1 bluestore(td/ok-to-upgrade/4) _read_fsid unparsable uuid 2026-02-21T11:37:28.325 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/4/keyring 2026-02-21T11:37:28.325 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:37:28.326 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-21T11:37:28.326 INFO:tasks.workunit.client.0.trial189.stdout:adding osd4 key to auth repository 2026-02-21T11:37:28.327 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:37:28.640 INFO:tasks.workunit.client.0.trial189.stdout:start osd.4 2026-02-21T11:37:28.641 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-21T11:37:28.641 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/4 --osd-journal=td/ok-to-upgrade/4/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:37:28.641 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:37:28.641 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:37:28.641 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:37:28.653 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:28.651+0000 7f2c3784d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:28.655 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:28.653+0000 7f2c3784d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:28.655 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:28.654+0000 7f2c3784d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:28.812 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:28.810+0000 7f2c3784d780 -1 Falling back to public interface 2026-02-21T11:37:28.865 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-21T11:37:28.865 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:37:28.865 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-21T11:37:28.866 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:37:28.866 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:37:28.866 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:28.866 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:37:28.866 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:37:28.867 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:28.867 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-21T11:37:29.087 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:37:29.115 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:29.113+0000 7f2c3784d780 -1 osd.4 0 log_to_monitors true 2026-02-21T11:37:30.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:37:30.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:30.089 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:37:30.090 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:37:30.090 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:30.090 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-21T11:37:30.312 INFO:tasks.workunit.client.0.trial189.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3559539331,v1:127.0.0.1:6835/3559539331] [v2:127.0.0.1:6836/3559539331,v1:127.0.0.1:6837/3559539331] exists,up fc759a60-6b17-44cb-a535-c04c614c87fe 2026-02-21T11:37:30.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:37:30.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:37:30.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:37:30.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:174: TEST_ok_to_upgrade_erasure_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:37:30.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:176: TEST_ok_to_upgrade_erasure_pool: run_osd td/ok-to-upgrade 5 --osd-mclock-skip-benchmark=true 2026-02-21T11:37:30.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:37:30.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:37:30.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-21T11:37:30.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:37:30.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/5 2026-02-21T11:37:30.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:37:30.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:37:30.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:37:30.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:37:30.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/5' 2026-02-21T11:37:30.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/5/journal' 2026-02-21T11:37:30.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:37:30.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:37:30.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:37:30.318 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:37:30.318 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:37:30.318 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:37:30.318 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:37:30.318 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:30.318 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:30.318 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:30.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:30.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:37:30.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:37:30.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:37:30.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:37:30.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:37:30.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:37:30.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:37:30.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:37:30.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:37:30.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:37:30.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:37:30.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/5 2026-02-21T11:37:30.320 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:37:30.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=69a99d64-bf75-4459-a56e-19ef85627608 2026-02-21T11:37:30.320 INFO:tasks.workunit.client.0.trial189.stdout:add osd5 69a99d64-bf75-4459-a56e-19ef85627608 2026-02-21T11:37:30.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 69a99d64-bf75-4459-a56e-19ef85627608' 2026-02-21T11:37:30.321 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:37:30.326 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD6mJlpgClXExAAPgdMiUgZvfbS5KEUs66CDg== 2026-02-21T11:37:30.326 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD6mJlpgClXExAAPgdMiUgZvfbS5KEUs66CDg=="}' 2026-02-21T11:37:30.326 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 69a99d64-bf75-4459-a56e-19ef85627608 -i td/ok-to-upgrade/5/new.json 2026-02-21T11:37:30.554 INFO:tasks.workunit.client.0.trial189.stdout:5 2026-02-21T11:37:30.562 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/5/new.json 2026-02-21T11:37:30.563 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/5 --osd-journal=td/ok-to-upgrade/5/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQD6mJlpgClXExAAPgdMiUgZvfbS5KEUs66CDg== --osd-uuid 69a99d64-bf75-4459-a56e-19ef85627608 2026-02-21T11:37:30.578 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:30.576+0000 7f0a80dd8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:30.579 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:30.578+0000 7f0a80dd8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:30.580 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:30.579+0000 7f0a80dd8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:30.581 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:30.579+0000 7f0a80dd8780 -1 bdev(0x3476c00 td/ok-to-upgrade/5/block) open stat got: (1) Operation not permitted 2026-02-21T11:37:30.581 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:30.579+0000 7f0a80dd8780 -1 bluestore(td/ok-to-upgrade/5) _read_fsid unparsable uuid 2026-02-21T11:37:30.903 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/5/keyring 2026-02-21T11:37:30.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:37:30.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-21T11:37:30.905 INFO:tasks.workunit.client.0.trial189.stdout:adding osd5 key to auth repository 2026-02-21T11:37:30.905 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:37:31.220 INFO:tasks.workunit.client.0.trial189.stdout:start osd.5 2026-02-21T11:37:31.221 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-21T11:37:31.221 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/5 --osd-journal=td/ok-to-upgrade/5/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:37:31.221 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:37:31.221 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:37:31.222 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:37:31.234 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:31.232+0000 7f1cb9a12780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:31.235 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:31.233+0000 7f1cb9a12780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:31.236 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:31.234+0000 7f1cb9a12780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:31.376 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:31.374+0000 7f1cb9a12780 -1 Falling back to public interface 2026-02-21T11:37:31.445 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-21T11:37:31.445 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:37:31.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-21T11:37:31.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:37:31.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:37:31.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:31.446 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:37:31.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:37:31.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:31.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-21T11:37:31.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:37:31.677 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:31.675+0000 7f1cb9a12780 -1 osd.5 0 log_to_monitors true 2026-02-21T11:37:32.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:37:32.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:32.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:37:32.673 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:37:32.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:32.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-21T11:37:32.900 INFO:tasks.workunit.client.0.trial189.stdout:osd.5 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3994964690,v1:127.0.0.1:6843/3994964690] [v2:127.0.0.1:6844/3994964690,v1:127.0.0.1:6845/3994964690] exists,up 69a99d64-bf75-4459-a56e-19ef85627608 2026-02-21T11:37:32.903 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:37:32.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:37:32.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:37:32.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:174: TEST_ok_to_upgrade_erasure_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:37:32.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:176: TEST_ok_to_upgrade_erasure_pool: run_osd td/ok-to-upgrade 6 --osd-mclock-skip-benchmark=true 2026-02-21T11:37:32.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:37:32.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:37:32.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-21T11:37:32.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:37:32.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/6 2026-02-21T11:37:32.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:37:32.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:37:32.905 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:37:32.905 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:37:32.905 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/6' 2026-02-21T11:37:32.905 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/6/journal' 2026-02-21T11:37:32.905 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:37:32.905 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:37:32.905 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:37:32.905 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:37:32.905 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:37:32.905 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:37:32.905 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:37:32.905 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:32.906 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:32.906 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:32.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:32.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:37:32.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:37:32.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:37:32.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:37:32.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:37:32.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:37:32.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:37:32.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:37:32.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:37:32.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:37:32.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:37:32.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/6 2026-02-21T11:37:32.907 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:37:32.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f5f06378-f2a4-45b1-a3ee-4968bff1dde4 2026-02-21T11:37:32.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 f5f06378-f2a4-45b1-a3ee-4968bff1dde4' 2026-02-21T11:37:32.907 INFO:tasks.workunit.client.0.trial189.stdout:add osd6 f5f06378-f2a4-45b1-a3ee-4968bff1dde4 2026-02-21T11:37:32.907 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:37:32.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD8mJlpzZNRNhAA8yCpvQBuO6No2FIGzSFW9A== 2026-02-21T11:37:32.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD8mJlpzZNRNhAA8yCpvQBuO6No2FIGzSFW9A=="}' 2026-02-21T11:37:32.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f5f06378-f2a4-45b1-a3ee-4968bff1dde4 -i td/ok-to-upgrade/6/new.json 2026-02-21T11:37:33.140 INFO:tasks.workunit.client.0.trial189.stdout:6 2026-02-21T11:37:33.147 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/6/new.json 2026-02-21T11:37:33.148 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/6 --osd-journal=td/ok-to-upgrade/6/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQD8mJlpzZNRNhAA8yCpvQBuO6No2FIGzSFW9A== --osd-uuid f5f06378-f2a4-45b1-a3ee-4968bff1dde4 2026-02-21T11:37:33.162 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:33.161+0000 7f9d12438780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:33.164 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:33.162+0000 7f9d12438780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:33.165 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:33.163+0000 7f9d12438780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:33.165 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:33.163+0000 7f9d12438780 -1 bdev(0x3ae4c00 td/ok-to-upgrade/6/block) open stat got: (1) Operation not permitted 2026-02-21T11:37:33.165 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:33.163+0000 7f9d12438780 -1 bluestore(td/ok-to-upgrade/6) _read_fsid unparsable uuid 2026-02-21T11:37:33.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/6/keyring 2026-02-21T11:37:33.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:37:33.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-21T11:37:33.497 INFO:tasks.workunit.client.0.trial189.stdout:adding osd6 key to auth repository 2026-02-21T11:37:33.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:37:33.824 INFO:tasks.workunit.client.0.trial189.stdout:start osd.6 2026-02-21T11:37:33.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-21T11:37:33.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/6 --osd-journal=td/ok-to-upgrade/6/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:37:33.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:37:33.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:37:33.826 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:37:33.838 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:33.836+0000 7fd44ef3d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:33.839 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:33.837+0000 7fd44ef3d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:33.840 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:33.838+0000 7fd44ef3d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:33.979 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:33.977+0000 7fd44ef3d780 -1 Falling back to public interface 2026-02-21T11:37:34.048 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-21T11:37:34.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:37:34.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-21T11:37:34.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:37:34.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:37:34.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:34.049 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:37:34.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:37:34.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:34.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-21T11:37:34.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:37:34.310 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:34.308+0000 7fd44ef3d780 -1 osd.6 0 log_to_monitors true 2026-02-21T11:37:35.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:37:35.275 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:35.275 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:37:35.276 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:37:35.276 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:35.276 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-21T11:37:35.496 INFO:tasks.workunit.client.0.trial189.stdout:osd.6 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/2720400559,v1:127.0.0.1:6851/2720400559] [v2:127.0.0.1:6852/2720400559,v1:127.0.0.1:6853/2720400559] exists,up f5f06378-f2a4-45b1-a3ee-4968bff1dde4 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:174: TEST_ok_to_upgrade_erasure_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:176: TEST_ok_to_upgrade_erasure_pool: run_osd td/ok-to-upgrade 7 --osd-mclock-skip-benchmark=true 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/7 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/7' 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/7/journal' 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:35.498 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:35.499 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:35.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:35.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:37:35.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:37:35.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:37:35.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:37:35.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:37:35.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:37:35.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:37:35.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:37:35.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:37:35.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:37:35.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:37:35.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/7 2026-02-21T11:37:35.500 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:37:35.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=71451cd3-aff8-4b78-b7d5-abb143649cbb 2026-02-21T11:37:35.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 71451cd3-aff8-4b78-b7d5-abb143649cbb' 2026-02-21T11:37:35.500 INFO:tasks.workunit.client.0.trial189.stdout:add osd7 71451cd3-aff8-4b78-b7d5-abb143649cbb 2026-02-21T11:37:35.500 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:37:35.509 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD/mJlp7QVMHhAA98BkfQ56Dz/4mjdz5XbVwA== 2026-02-21T11:37:35.509 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD/mJlp7QVMHhAA98BkfQ56Dz/4mjdz5XbVwA=="}' 2026-02-21T11:37:35.509 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 71451cd3-aff8-4b78-b7d5-abb143649cbb -i td/ok-to-upgrade/7/new.json 2026-02-21T11:37:35.737 INFO:tasks.workunit.client.0.trial189.stdout:7 2026-02-21T11:37:35.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/7/new.json 2026-02-21T11:37:35.745 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/7 --osd-journal=td/ok-to-upgrade/7/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQD/mJlp7QVMHhAA98BkfQ56Dz/4mjdz5XbVwA== --osd-uuid 71451cd3-aff8-4b78-b7d5-abb143649cbb 2026-02-21T11:37:35.760 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:35.758+0000 7f92f1448780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:35.761 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:35.760+0000 7f92f1448780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:35.762 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:35.761+0000 7f92f1448780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:35.763 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:35.761+0000 7f92f1448780 -1 bdev(0x2d88c00 td/ok-to-upgrade/7/block) open stat got: (1) Operation not permitted 2026-02-21T11:37:35.763 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:35.761+0000 7f92f1448780 -1 bluestore(td/ok-to-upgrade/7) _read_fsid unparsable uuid 2026-02-21T11:37:36.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/7/keyring 2026-02-21T11:37:36.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:37:36.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2026-02-21T11:37:36.096 INFO:tasks.workunit.client.0.trial189.stdout:adding osd7 key to auth repository 2026-02-21T11:37:36.097 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:37:36.413 INFO:tasks.workunit.client.0.trial189.stdout:start osd.7 2026-02-21T11:37:36.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2026-02-21T11:37:36.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/7 --osd-journal=td/ok-to-upgrade/7/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:37:36.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:37:36.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:37:36.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:37:36.426 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:36.425+0000 7feecfe2b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:36.428 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:36.426+0000 7feecfe2b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:36.429 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:36.427+0000 7feecfe2b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:36.580 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:36.578+0000 7feecfe2b780 -1 Falling back to public interface 2026-02-21T11:37:36.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2026-02-21T11:37:36.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:37:36.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2026-02-21T11:37:36.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:37:36.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:37:36.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:36.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:37:36.644 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:37:36.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:36.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-21T11:37:36.867 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:37:36.933 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:36.931+0000 7feecfe2b780 -1 osd.7 0 log_to_monitors true 2026-02-21T11:37:37.868 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:37:37.868 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:37.869 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:37:37.869 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:37:37.869 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:37.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-21T11:37:38.094 INFO:tasks.workunit.client.0.trial189.stdout:osd.7 up in weight 1 up_from 38 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3398349462,v1:127.0.0.1:6859/3398349462] [v2:127.0.0.1:6860/3398349462,v1:127.0.0.1:6861/3398349462] exists,up 71451cd3-aff8-4b78-b7d5-abb143649cbb 2026-02-21T11:37:38.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:37:38.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:37:38.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:37:38.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:174: TEST_ok_to_upgrade_erasure_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:37:38.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:176: TEST_ok_to_upgrade_erasure_pool: run_osd td/ok-to-upgrade 8 --osd-mclock-skip-benchmark=true 2026-02-21T11:37:38.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:37:38.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:37:38.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2026-02-21T11:37:38.099 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:37:38.099 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/8 2026-02-21T11:37:38.099 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:37:38.099 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:37:38.099 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:37:38.099 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:37:38.099 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/8' 2026-02-21T11:37:38.099 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/8/journal' 2026-02-21T11:37:38.099 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:37:38.099 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:37:38.099 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:37:38.099 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:37:38.100 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:37:38.100 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:37:38.100 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:37:38.100 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:38.100 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:38.100 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:38.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:38.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:37:38.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:37:38.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:37:38.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:37:38.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:37:38.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:37:38.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:37:38.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:37:38.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:37:38.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:37:38.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:37:38.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/8 2026-02-21T11:37:38.101 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:37:38.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=544f5246-0ec0-4d8d-8f2c-d0aa4d23452b 2026-02-21T11:37:38.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 544f5246-0ec0-4d8d-8f2c-d0aa4d23452b' 2026-02-21T11:37:38.102 INFO:tasks.workunit.client.0.trial189.stdout:add osd8 544f5246-0ec0-4d8d-8f2c-d0aa4d23452b 2026-02-21T11:37:38.102 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:37:38.106 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQACmZlpSYBIBhAAKyMC3P6+lsziZFeShZOoDQ== 2026-02-21T11:37:38.106 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQACmZlpSYBIBhAAKyMC3P6+lsziZFeShZOoDQ=="}' 2026-02-21T11:37:38.106 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 544f5246-0ec0-4d8d-8f2c-d0aa4d23452b -i td/ok-to-upgrade/8/new.json 2026-02-21T11:37:38.334 INFO:tasks.workunit.client.0.trial189.stdout:8 2026-02-21T11:37:38.341 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/8/new.json 2026-02-21T11:37:38.342 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/8 --osd-journal=td/ok-to-upgrade/8/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQACmZlpSYBIBhAAKyMC3P6+lsziZFeShZOoDQ== --osd-uuid 544f5246-0ec0-4d8d-8f2c-d0aa4d23452b 2026-02-21T11:37:38.356 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:38.354+0000 7f8248862780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:38.357 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:38.356+0000 7f8248862780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:38.358 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:38.356+0000 7f8248862780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:38.358 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:38.357+0000 7f8248862780 -1 bdev(0x2f86c00 td/ok-to-upgrade/8/block) open stat got: (1) Operation not permitted 2026-02-21T11:37:38.358 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:38.357+0000 7f8248862780 -1 bluestore(td/ok-to-upgrade/8) _read_fsid unparsable uuid 2026-02-21T11:37:38.680 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/8/keyring 2026-02-21T11:37:38.680 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:37:38.680 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2026-02-21T11:37:38.681 INFO:tasks.workunit.client.0.trial189.stdout:adding osd8 key to auth repository 2026-02-21T11:37:38.681 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:37:38.995 INFO:tasks.workunit.client.0.trial189.stdout:start osd.8 2026-02-21T11:37:38.995 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2026-02-21T11:37:38.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/8 --osd-journal=td/ok-to-upgrade/8/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:37:38.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:37:38.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:37:38.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:37:39.008 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:39.006+0000 7fd139234780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:39.009 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:39.008+0000 7fd139234780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:39.010 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:39.008+0000 7fd139234780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:39.170 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:39.168+0000 7fd139234780 -1 Falling back to public interface 2026-02-21T11:37:39.220 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2026-02-21T11:37:39.220 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:37:39.220 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2026-02-21T11:37:39.220 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:37:39.220 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:37:39.220 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:39.220 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:37:39.220 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:37:39.220 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:39.220 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-21T11:37:39.443 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:37:39.543 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:39.541+0000 7fd139234780 -1 osd.8 0 log_to_monitors true 2026-02-21T11:37:40.444 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:37:40.444 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:40.445 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:37:40.445 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:37:40.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:40.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-21T11:37:40.666 INFO:tasks.workunit.client.0.trial189.stdout:osd.8 up in weight 1 up_from 42 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/1275510998,v1:127.0.0.1:6867/1275510998] [v2:127.0.0.1:6868/1275510998,v1:127.0.0.1:6869/1275510998] exists,up 544f5246-0ec0-4d8d-8f2c-d0aa4d23452b 2026-02-21T11:37:40.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:37:40.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:37:40.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:37:40.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:174: TEST_ok_to_upgrade_erasure_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:37:40.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:176: TEST_ok_to_upgrade_erasure_pool: run_osd td/ok-to-upgrade 9 --osd-mclock-skip-benchmark=true 2026-02-21T11:37:40.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:37:40.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:37:40.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2026-02-21T11:37:40.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:37:40.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/9 2026-02-21T11:37:40.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:37:40.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:37:40.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:37:40.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:37:40.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/9' 2026-02-21T11:37:40.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/9/journal' 2026-02-21T11:37:40.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:37:40.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:37:40.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:37:40.671 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:37:40.671 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:37:40.671 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:37:40.671 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:37:40.671 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:37:40.671 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:37:40.671 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:40.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:37:40.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:37:40.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:37:40.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:37:40.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:37:40.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:37:40.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:37:40.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:37:40.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:37:40.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:37:40.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:37:40.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:37:40.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/9 2026-02-21T11:37:40.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:37:40.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7b343da2-55b6-4f26-9a3f-5cfedc7cfe3c 2026-02-21T11:37:40.673 INFO:tasks.workunit.client.0.trial189.stdout:add osd9 7b343da2-55b6-4f26-9a3f-5cfedc7cfe3c 2026-02-21T11:37:40.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 7b343da2-55b6-4f26-9a3f-5cfedc7cfe3c' 2026-02-21T11:37:40.674 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:37:40.678 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAEmZlpcPhiKBAAvvS2ZRF1aZ2CZAgDFO3tWA== 2026-02-21T11:37:40.679 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAEmZlpcPhiKBAAvvS2ZRF1aZ2CZAgDFO3tWA=="}' 2026-02-21T11:37:40.679 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7b343da2-55b6-4f26-9a3f-5cfedc7cfe3c -i td/ok-to-upgrade/9/new.json 2026-02-21T11:37:40.906 INFO:tasks.workunit.client.0.trial189.stdout:9 2026-02-21T11:37:40.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/9/new.json 2026-02-21T11:37:40.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/9 --osd-journal=td/ok-to-upgrade/9/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQAEmZlpcPhiKBAAvvS2ZRF1aZ2CZAgDFO3tWA== --osd-uuid 7b343da2-55b6-4f26-9a3f-5cfedc7cfe3c 2026-02-21T11:37:40.929 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:40.927+0000 7f1832849780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:40.931 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:40.929+0000 7f1832849780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:40.931 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:40.930+0000 7f1832849780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:40.932 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:40.930+0000 7f1832849780 -1 bdev(0x2b0cc00 td/ok-to-upgrade/9/block) open stat got: (1) Operation not permitted 2026-02-21T11:37:40.932 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:40.930+0000 7f1832849780 -1 bluestore(td/ok-to-upgrade/9) _read_fsid unparsable uuid 2026-02-21T11:37:41.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/9/keyring 2026-02-21T11:37:41.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:37:41.254 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2026-02-21T11:37:41.254 INFO:tasks.workunit.client.0.trial189.stdout:adding osd9 key to auth repository 2026-02-21T11:37:41.255 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:37:41.574 INFO:tasks.workunit.client.0.trial189.stdout:start osd.9 2026-02-21T11:37:41.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2026-02-21T11:37:41.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/9 --osd-journal=td/ok-to-upgrade/9/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:37:41.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:37:41.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:37:41.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:37:41.588 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:41.586+0000 7fd03af3d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:41.589 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:41.587+0000 7fd03af3d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:41.590 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:41.588+0000 7fd03af3d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:37:41.777 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:41.776+0000 7fd03af3d780 -1 Falling back to public interface 2026-02-21T11:37:41.799 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2026-02-21T11:37:41.799 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:37:41.799 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-21T11:37:41.799 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:37:41.799 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:37:41.799 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:41.800 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:37:41.800 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:37:41.800 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:41.801 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-21T11:37:42.023 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:37:42.179 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:37:42.177+0000 7fd03af3d780 -1 osd.9 0 log_to_monitors true 2026-02-21T11:37:43.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:37:43.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:37:43.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:37:43.025 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:37:43.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:37:43.026 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-21T11:37:43.248 INFO:tasks.workunit.client.0.trial189.stdout:osd.9 up in weight 1 up_from 46 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/2523959361,v1:127.0.0.1:6875/2523959361] [v2:127.0.0.1:6876/2523959361,v1:127.0.0.1:6877/2523959361] exists,up 7b343da2-55b6-4f26-9a3f-5cfedc7cfe3c 2026-02-21T11:37:43.249 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:37:43.249 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:37:43.249 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:37:43.249 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:179: TEST_ok_to_upgrade_erasure_pool: ceph osd erasure-code-profile set ec-profile m=3 k=5 crush-failure-domain=osd 2026-02-21T11:37:43.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:180: TEST_ok_to_upgrade_erasure_pool: ceph osd pool create ec erasure ec-profile 2026-02-21T11:37:43.883 INFO:tasks.workunit.client.0.trial189.stderr:pool 'ec' already exists 2026-02-21T11:37:43.891 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:181: TEST_ok_to_upgrade_erasure_pool: ceph osd pool set ec min_size 5 2026-02-21T11:37:44.229 INFO:tasks.workunit.client.0.trial189.stderr:set pool 1 min_size to 5 2026-02-21T11:37:44.250 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:182: TEST_ok_to_upgrade_erasure_pool: sleep 5 2026-02-21T11:37:49.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:184: TEST_ok_to_upgrade_erasure_pool: wait_for_clean 2026-02-21T11:37:49.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:37:49.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:37:49.254 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:37:49.254 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:37:49.254 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:37:49.254 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:37:49.254 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:37:49.254 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:37:49.254 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:37:49.336 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:37:49.336 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:37:49.336 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:37:49.336 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:37:49.336 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:37:49.336 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:37:49.557 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:37:49.557 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:37:49.557 INFO:tasks.workunit.client.0.trial189.stderr:2 2026-02-21T11:37:49.557 INFO:tasks.workunit.client.0.trial189.stderr:3 2026-02-21T11:37:49.557 INFO:tasks.workunit.client.0.trial189.stderr:4 2026-02-21T11:37:49.557 INFO:tasks.workunit.client.0.trial189.stderr:5 2026-02-21T11:37:49.557 INFO:tasks.workunit.client.0.trial189.stderr:6 2026-02-21T11:37:49.557 INFO:tasks.workunit.client.0.trial189.stderr:7 2026-02-21T11:37:49.558 INFO:tasks.workunit.client.0.trial189.stderr:8 2026-02-21T11:37:49.558 INFO:tasks.workunit.client.0.trial189.stderr:9' 2026-02-21T11:37:49.558 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:37:49.558 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:49.558 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:37:49.621 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836488 2026-02-21T11:37:49.621 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836488 2026-02-21T11:37:49.621 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836488' 2026-02-21T11:37:49.621 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:49.621 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:37:49.683 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=38654705672 2026-02-21T11:37:49.683 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 38654705672 2026-02-21T11:37:49.683 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836488 1-38654705672' 2026-02-21T11:37:49.684 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:49.684 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-21T11:37:49.746 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=60129542151 2026-02-21T11:37:49.746 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 60129542151 2026-02-21T11:37:49.746 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836488 1-38654705672 2-60129542151' 2026-02-21T11:37:49.746 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:49.746 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-21T11:37:49.808 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=81604378632 2026-02-21T11:37:49.808 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 81604378632 2026-02-21T11:37:49.808 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836488 1-38654705672 2-60129542151 3-81604378632' 2026-02-21T11:37:49.809 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:49.809 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-21T11:37:49.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=98784247814 2026-02-21T11:37:49.871 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 98784247814 2026-02-21T11:37:49.871 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836488 1-38654705672 2-60129542151 3-81604378632 4-98784247814' 2026-02-21T11:37:49.871 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:49.871 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-21T11:37:49.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=120259084295 2026-02-21T11:37:49.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 120259084295 2026-02-21T11:37:49.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836488 1-38654705672 2-60129542151 3-81604378632 4-98784247814 5-120259084295' 2026-02-21T11:37:49.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:49.936 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-21T11:37:49.998 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=141733920773 2026-02-21T11:37:49.998 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 141733920773 2026-02-21T11:37:49.998 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836488 1-38654705672 2-60129542151 3-81604378632 4-98784247814 5-120259084295 6-141733920773' 2026-02-21T11:37:49.998 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:49.998 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-21T11:37:50.059 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=163208757254 2026-02-21T11:37:50.059 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 163208757254 2026-02-21T11:37:50.059 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836488 1-38654705672 2-60129542151 3-81604378632 4-98784247814 5-120259084295 6-141733920773 7-163208757254' 2026-02-21T11:37:50.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:50.060 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-21T11:37:50.124 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=180388626437 2026-02-21T11:37:50.124 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 180388626437 2026-02-21T11:37:50.124 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836488 1-38654705672 2-60129542151 3-81604378632 4-98784247814 5-120259084295 6-141733920773 7-163208757254 8-180388626437' 2026-02-21T11:37:50.124 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:50.125 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-21T11:37:50.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=197568495620 2026-02-21T11:37:50.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 197568495620 2026-02-21T11:37:50.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836488 1-38654705672 2-60129542151 3-81604378632 4-98784247814 5-120259084295 6-141733920773 7-163208757254 8-180388626437 9-197568495620' 2026-02-21T11:37:50.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:37:50.187 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836488 2026-02-21T11:37:50.187 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:37:50.188 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:37:50.189 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836488 2026-02-21T11:37:50.189 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:37:50.190 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836488 2026-02-21T11:37:50.190 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-21T11:37:50.190 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836488 2026-02-21T11:37:50.190 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:37:50.416 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-21T11:37:50.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:37:50.417 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-38654705672 2026-02-21T11:37:50.417 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:37:50.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:37:50.418 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-38654705672 2026-02-21T11:37:50.418 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:37:50.419 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=38654705672 2026-02-21T11:37:50.419 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2026-02-21T11:37:50.420 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 38654705672 2026-02-21T11:37:50.420 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:37:50.639 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 38654705672 -lt 38654705672 2026-02-21T11:37:50.640 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:37:50.640 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 2-60129542151 2026-02-21T11:37:50.640 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:37:50.641 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=2 2026-02-21T11:37:50.641 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 2-60129542151 2026-02-21T11:37:50.641 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:37:50.642 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=60129542151 2026-02-21T11:37:50.643 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.2 seq 60129542151 2026-02-21T11:37:50.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2026-02-21T11:37:50.643 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T11:37:50.866 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 60129542151 -lt 60129542151 2026-02-21T11:37:50.866 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:37:50.867 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 3-81604378632 2026-02-21T11:37:50.867 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:37:50.868 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=3 2026-02-21T11:37:50.868 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 3-81604378632 2026-02-21T11:37:50.868 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:37:50.869 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=81604378632 2026-02-21T11:37:50.869 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.3 seq 81604378632' 2026-02-21T11:37:50.870 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.3 seq 81604378632 2026-02-21T11:37:50.870 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-21T11:37:51.093 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 81604378632 -lt 81604378632 2026-02-21T11:37:51.093 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:37:51.093 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 4-98784247814 2026-02-21T11:37:51.093 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:37:51.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=4 2026-02-21T11:37:51.094 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 4-98784247814 2026-02-21T11:37:51.095 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:37:51.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=98784247814 2026-02-21T11:37:51.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.4 seq 98784247814' 2026-02-21T11:37:51.096 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.4 seq 98784247814 2026-02-21T11:37:51.096 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-21T11:37:51.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 98784247814 -lt 98784247814 2026-02-21T11:37:51.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:37:51.316 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 5-120259084295 2026-02-21T11:37:51.317 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:37:51.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=5 2026-02-21T11:37:51.318 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 5-120259084295 2026-02-21T11:37:51.318 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:37:51.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=120259084295 2026-02-21T11:37:51.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.5 seq 120259084295' 2026-02-21T11:37:51.320 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.5 seq 120259084295 2026-02-21T11:37:51.320 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-21T11:37:51.540 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 120259084295 -lt 120259084295 2026-02-21T11:37:51.541 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:37:51.541 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 6-141733920773 2026-02-21T11:37:51.541 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:37:51.542 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=6 2026-02-21T11:37:51.542 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 6-141733920773 2026-02-21T11:37:51.542 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:37:51.543 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=141733920773 2026-02-21T11:37:51.544 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.6 seq 141733920773' 2026-02-21T11:37:51.544 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.6 seq 141733920773 2026-02-21T11:37:51.544 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-21T11:37:51.766 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 141733920773 -lt 141733920773 2026-02-21T11:37:51.766 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:37:51.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 7-163208757254 2026-02-21T11:37:51.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:37:51.768 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=7 2026-02-21T11:37:51.768 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 7-163208757254 2026-02-21T11:37:51.768 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:37:51.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=163208757254 2026-02-21T11:37:51.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.7 seq 163208757254' 2026-02-21T11:37:51.769 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.7 seq 163208757254 2026-02-21T11:37:51.770 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-21T11:37:51.995 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 163208757254 -lt 163208757254 2026-02-21T11:37:51.995 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:37:51.996 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 8-180388626437 2026-02-21T11:37:51.996 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:37:51.997 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=8 2026-02-21T11:37:51.997 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 8-180388626437 2026-02-21T11:37:51.997 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:37:51.998 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=180388626437 2026-02-21T11:37:51.999 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.8 seq 180388626437' 2026-02-21T11:37:51.999 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.8 seq 180388626437 2026-02-21T11:37:51.999 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-21T11:37:52.224 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 180388626437 -lt 180388626437 2026-02-21T11:37:52.224 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:37:52.224 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 9-197568495620 2026-02-21T11:37:52.224 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:37:52.225 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=9 2026-02-21T11:37:52.226 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 9-197568495620 2026-02-21T11:37:52.226 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:37:52.227 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=197568495620 2026-02-21T11:37:52.227 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.9 seq 197568495620' 2026-02-21T11:37:52.227 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.9 seq 197568495620 2026-02-21T11:37:52.227 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-21T11:37:52.452 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 197568495620 -lt 197568495620 2026-02-21T11:37:52.453 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:37:52.453 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:37:52.454 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:37:52.768 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-21T11:37:52.768 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:37:52.768 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:37:52.768 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:37:52.768 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:37:52.768 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:37:52.768 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:37:52.769 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:37:52.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-21T11:37:52.988 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:37:52.988 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:37:52.988 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:37:53.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-21T11:37:53.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:37:53.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:37:53.317 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:187: TEST_ok_to_upgrade_erasure_pool: local exp_osds_upgradable=3 2026-02-21T11:37:53.317 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:188: TEST_ok_to_upgrade_erasure_pool: ceph osd tree 2026-02-21T11:37:53.317 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:188: TEST_ok_to_upgrade_erasure_pool: grep host 2026-02-21T11:37:53.318 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:188: TEST_ok_to_upgrade_erasure_pool: awk '{ print $4 }' 2026-02-21T11:37:53.539 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:188: TEST_ok_to_upgrade_erasure_pool: local crush_bucket=trial189 2026-02-21T11:37:53.539 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:189: TEST_ok_to_upgrade_erasure_pool: ceph osd ok-to-upgrade trial189 01.2.3-1234-g1234deed --format=json 2026-02-21T11:37:53.761 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:189: TEST_ok_to_upgrade_erasure_pool: local 'res= 2026-02-21T11:37:53.761 INFO:tasks.workunit.client.0.trial189.stderr:{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[7,2,4],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:37:53.762 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:193: TEST_ok_to_upgrade_erasure_pool: jq .all_osds_upgraded 2026-02-21T11:37:53.762 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:193: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[7,2,4],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:37:53.764 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:193: TEST_ok_to_upgrade_erasure_pool: test false = false 2026-02-21T11:37:53.764 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:194: TEST_ok_to_upgrade_erasure_pool: jq .ok_to_upgrade 2026-02-21T11:37:53.764 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:194: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[7,2,4],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:37:53.766 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:194: TEST_ok_to_upgrade_erasure_pool: test true = true 2026-02-21T11:37:53.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:195: TEST_ok_to_upgrade_erasure_pool: jq '.osds_ok_to_upgrade | length' 2026-02-21T11:37:53.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:195: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[7,2,4],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:37:53.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:195: TEST_ok_to_upgrade_erasure_pool: bc 2026-02-21T11:37:53.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:195: TEST_ok_to_upgrade_erasure_pool: local num_osds_upgradable=3 2026-02-21T11:37:53.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:196: TEST_ok_to_upgrade_erasure_pool: test 3 = 3 2026-02-21T11:37:53.769 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:197: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[7,2,4],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:37:53.770 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:197: TEST_ok_to_upgrade_erasure_pool: jq '.osds_upgraded | length' 2026-02-21T11:37:53.770 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:197: TEST_ok_to_upgrade_erasure_pool: bc 2026-02-21T11:37:53.772 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:197: TEST_ok_to_upgrade_erasure_pool: local num_osds_upgraded=0 2026-02-21T11:37:53.772 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:198: TEST_ok_to_upgrade_erasure_pool: test 0 -eq 0 2026-02-21T11:37:53.772 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:203: TEST_ok_to_upgrade_erasure_pool: local max=3 2026-02-21T11:37:53.772 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:204: TEST_ok_to_upgrade_erasure_pool: crush_bucket=osd.0 2026-02-21T11:37:53.772 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:205: TEST_ok_to_upgrade_erasure_pool: ceph osd ok-to-upgrade osd.0 01.2.3-1234-g1234deed 3 --format=json 2026-02-21T11:37:53.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:205: TEST_ok_to_upgrade_erasure_pool: res=' 2026-02-21T11:37:53.997 INFO:tasks.workunit.client.0.trial189.stderr:{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[0],"osds_ok_to_upgrade":[0,7,2],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:37:53.998 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:206: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[0],"osds_ok_to_upgrade":[0,7,2],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:37:53.998 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:206: TEST_ok_to_upgrade_erasure_pool: jq .all_osds_upgraded 2026-02-21T11:37:53.999 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:206: TEST_ok_to_upgrade_erasure_pool: test false = false 2026-02-21T11:37:54.000 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:207: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[0],"osds_ok_to_upgrade":[0,7,2],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:37:54.000 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:207: TEST_ok_to_upgrade_erasure_pool: jq .ok_to_upgrade 2026-02-21T11:37:54.002 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:207: TEST_ok_to_upgrade_erasure_pool: test true = true 2026-02-21T11:37:54.003 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:208: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[0],"osds_ok_to_upgrade":[0,7,2],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:37:54.003 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:208: TEST_ok_to_upgrade_erasure_pool: jq '.osds_ok_to_upgrade | length' 2026-02-21T11:37:54.003 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:208: TEST_ok_to_upgrade_erasure_pool: bc 2026-02-21T11:37:54.005 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:208: TEST_ok_to_upgrade_erasure_pool: num_osds_upgradable=3 2026-02-21T11:37:54.006 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:209: TEST_ok_to_upgrade_erasure_pool: test 3 = 3 2026-02-21T11:37:54.006 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:210: TEST_ok_to_upgrade_erasure_pool: test 3 = 3 2026-02-21T11:37:54.006 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:211: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[0],"osds_ok_to_upgrade":[0,7,2],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:37:54.006 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:211: TEST_ok_to_upgrade_erasure_pool: jq '.osds_upgraded | length' 2026-02-21T11:37:54.006 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:211: TEST_ok_to_upgrade_erasure_pool: bc 2026-02-21T11:37:54.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:211: TEST_ok_to_upgrade_erasure_pool: num_osds_upgraded=0 2026-02-21T11:37:54.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:212: TEST_ok_to_upgrade_erasure_pool: test 0 -eq 0 2026-02-21T11:37:54.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:215: TEST_ok_to_upgrade_erasure_pool: ceph osd pool set ec min_size 6 2026-02-21T11:37:54.317 INFO:tasks.workunit.client.0.trial189.stderr:set pool 1 min_size to 6 2026-02-21T11:37:54.336 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:216: TEST_ok_to_upgrade_erasure_pool: sleep 5 2026-02-21T11:37:59.337 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:217: TEST_ok_to_upgrade_erasure_pool: wait_for_clean 2026-02-21T11:37:59.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:37:59.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:37:59.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:37:59.339 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:37:59.339 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:37:59.339 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:37:59.339 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:37:59.339 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:37:59.339 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:37:59.424 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:37:59.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:37:59.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:37:59.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:37:59.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:37:59.425 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:37:59.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:37:59.645 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:37:59.645 INFO:tasks.workunit.client.0.trial189.stderr:2 2026-02-21T11:37:59.645 INFO:tasks.workunit.client.0.trial189.stderr:3 2026-02-21T11:37:59.645 INFO:tasks.workunit.client.0.trial189.stderr:4 2026-02-21T11:37:59.645 INFO:tasks.workunit.client.0.trial189.stderr:5 2026-02-21T11:37:59.645 INFO:tasks.workunit.client.0.trial189.stderr:6 2026-02-21T11:37:59.645 INFO:tasks.workunit.client.0.trial189.stderr:7 2026-02-21T11:37:59.645 INFO:tasks.workunit.client.0.trial189.stderr:8 2026-02-21T11:37:59.645 INFO:tasks.workunit.client.0.trial189.stderr:9' 2026-02-21T11:37:59.645 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:37:59.645 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:59.645 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:37:59.706 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836492 2026-02-21T11:37:59.706 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836492 2026-02-21T11:37:59.706 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836492' 2026-02-21T11:37:59.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:59.707 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:37:59.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=38654705676 2026-02-21T11:37:59.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 38654705676 2026-02-21T11:37:59.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836492 1-38654705676' 2026-02-21T11:37:59.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:59.769 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-21T11:37:59.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=60129542155 2026-02-21T11:37:59.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 60129542155 2026-02-21T11:37:59.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-60129542155' 2026-02-21T11:37:59.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:59.832 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-21T11:37:59.896 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=81604378636 2026-02-21T11:37:59.896 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 81604378636 2026-02-21T11:37:59.896 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-60129542155 3-81604378636' 2026-02-21T11:37:59.896 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:59.896 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-21T11:37:59.957 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=98784247818 2026-02-21T11:37:59.957 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 98784247818 2026-02-21T11:37:59.957 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-60129542155 3-81604378636 4-98784247818' 2026-02-21T11:37:59.958 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:37:59.958 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-21T11:38:00.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=120259084299 2026-02-21T11:38:00.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 120259084299 2026-02-21T11:38:00.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-60129542155 3-81604378636 4-98784247818 5-120259084299' 2026-02-21T11:38:00.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:00.026 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-21T11:38:00.085 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=141733920777 2026-02-21T11:38:00.085 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 141733920777 2026-02-21T11:38:00.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-60129542155 3-81604378636 4-98784247818 5-120259084299 6-141733920777' 2026-02-21T11:38:00.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:00.086 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-21T11:38:00.149 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=163208757258 2026-02-21T11:38:00.149 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 163208757258 2026-02-21T11:38:00.149 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-60129542155 3-81604378636 4-98784247818 5-120259084299 6-141733920777 7-163208757258' 2026-02-21T11:38:00.149 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:00.149 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-21T11:38:00.211 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=180388626441 2026-02-21T11:38:00.211 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 180388626441 2026-02-21T11:38:00.211 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-60129542155 3-81604378636 4-98784247818 5-120259084299 6-141733920777 7-163208757258 8-180388626441' 2026-02-21T11:38:00.211 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:00.211 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-21T11:38:00.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=197568495624 2026-02-21T11:38:00.275 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 197568495624 2026-02-21T11:38:00.275 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836492 1-38654705676 2-60129542155 3-81604378636 4-98784247818 5-120259084299 6-141733920777 7-163208757258 8-180388626441 9-197568495624' 2026-02-21T11:38:00.275 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:00.275 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836492 2026-02-21T11:38:00.275 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:00.275 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:38:00.276 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836492 2026-02-21T11:38:00.276 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:00.277 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836492 2026-02-21T11:38:00.277 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836492 2026-02-21T11:38:00.278 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2026-02-21T11:38:00.278 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:38:00.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836492 -lt 21474836492 2026-02-21T11:38:00.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:00.499 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-38654705676 2026-02-21T11:38:00.500 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:00.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:38:00.500 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-38654705676 2026-02-21T11:38:00.501 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:00.501 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=38654705676 2026-02-21T11:38:00.502 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2026-02-21T11:38:00.502 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 38654705676 2026-02-21T11:38:00.502 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:38:00.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 38654705676 -lt 38654705676 2026-02-21T11:38:00.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:00.725 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 2-60129542155 2026-02-21T11:38:00.725 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:00.725 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=2 2026-02-21T11:38:00.725 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 2-60129542155 2026-02-21T11:38:00.726 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:00.727 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=60129542155 2026-02-21T11:38:00.727 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2026-02-21T11:38:00.727 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.2 seq 60129542155 2026-02-21T11:38:00.727 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T11:38:00.950 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 60129542155 -lt 60129542155 2026-02-21T11:38:00.951 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:00.951 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 3-81604378636 2026-02-21T11:38:00.951 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:00.952 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=3 2026-02-21T11:38:00.952 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 3-81604378636 2026-02-21T11:38:00.953 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:00.953 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=81604378636 2026-02-21T11:38:00.953 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.3 seq 81604378636' 2026-02-21T11:38:00.954 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.3 seq 81604378636 2026-02-21T11:38:00.954 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-21T11:38:01.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 81604378636 -lt 81604378636 2026-02-21T11:38:01.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:01.176 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 4-98784247818 2026-02-21T11:38:01.176 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:01.177 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=4 2026-02-21T11:38:01.177 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 4-98784247818 2026-02-21T11:38:01.177 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:01.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=98784247818 2026-02-21T11:38:01.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.4 seq 98784247818' 2026-02-21T11:38:01.179 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.4 seq 98784247818 2026-02-21T11:38:01.179 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-21T11:38:01.403 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 98784247818 -lt 98784247818 2026-02-21T11:38:01.403 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:01.404 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 5-120259084299 2026-02-21T11:38:01.404 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:01.404 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=5 2026-02-21T11:38:01.405 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 5-120259084299 2026-02-21T11:38:01.405 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:01.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=120259084299 2026-02-21T11:38:01.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.5 seq 120259084299' 2026-02-21T11:38:01.406 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.5 seq 120259084299 2026-02-21T11:38:01.406 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-21T11:38:01.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 120259084299 -lt 120259084299 2026-02-21T11:38:01.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:01.629 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 6-141733920777 2026-02-21T11:38:01.629 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:01.630 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=6 2026-02-21T11:38:01.631 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 6-141733920777 2026-02-21T11:38:01.631 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:01.632 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=141733920777 2026-02-21T11:38:01.632 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.6 seq 141733920777' 2026-02-21T11:38:01.632 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.6 seq 141733920777 2026-02-21T11:38:01.632 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-21T11:38:01.860 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 141733920777 -lt 141733920777 2026-02-21T11:38:01.860 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:01.860 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 7-163208757258 2026-02-21T11:38:01.860 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:01.861 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=7 2026-02-21T11:38:01.862 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 7-163208757258 2026-02-21T11:38:01.862 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:01.863 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=163208757258 2026-02-21T11:38:01.863 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.7 seq 163208757258' 2026-02-21T11:38:01.863 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.7 seq 163208757258 2026-02-21T11:38:01.863 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-21T11:38:02.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 163208757258 -lt 163208757258 2026-02-21T11:38:02.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:02.087 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 8-180388626441 2026-02-21T11:38:02.087 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:02.088 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=8 2026-02-21T11:38:02.088 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 8-180388626441 2026-02-21T11:38:02.088 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:02.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=180388626441 2026-02-21T11:38:02.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.8 seq 180388626441' 2026-02-21T11:38:02.090 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.8 seq 180388626441 2026-02-21T11:38:02.090 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-21T11:38:02.313 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 180388626442 -lt 180388626441 2026-02-21T11:38:02.314 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:02.314 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 9-197568495624 2026-02-21T11:38:02.314 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:02.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=9 2026-02-21T11:38:02.315 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 9-197568495624 2026-02-21T11:38:02.315 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:02.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=197568495624 2026-02-21T11:38:02.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.9 seq 197568495624' 2026-02-21T11:38:02.317 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.9 seq 197568495624 2026-02-21T11:38:02.317 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-21T11:38:02.543 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 197568495624 -lt 197568495624 2026-02-21T11:38:02.544 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:38:02.544 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:38:02.544 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:38:02.861 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-21T11:38:02.861 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:38:02.861 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:38:02.861 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:38:02.861 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:38:02.861 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:38:02.861 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:38:02.861 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:38:03.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-21T11:38:03.084 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:38:03.084 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:38:03.084 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:38:03.429 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-21T11:38:03.429 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:38:03.429 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:38:03.429 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:218: TEST_ok_to_upgrade_erasure_pool: exp_osds_upgradable=2 2026-02-21T11:38:03.429 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:219: TEST_ok_to_upgrade_erasure_pool: ceph osd tree 2026-02-21T11:38:03.429 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:219: TEST_ok_to_upgrade_erasure_pool: grep host 2026-02-21T11:38:03.429 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:219: TEST_ok_to_upgrade_erasure_pool: awk '{ print $4 }' 2026-02-21T11:38:03.653 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:219: TEST_ok_to_upgrade_erasure_pool: crush_bucket=trial189 2026-02-21T11:38:03.654 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:220: TEST_ok_to_upgrade_erasure_pool: ceph osd ok-to-upgrade trial189 01.2.3-1234-g1234deed --format=json 2026-02-21T11:38:03.881 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:220: TEST_ok_to_upgrade_erasure_pool: res=' 2026-02-21T11:38:03.881 INFO:tasks.workunit.client.0.trial189.stderr:{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[7,2],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:03.882 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:222: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[7,2],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:03.882 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:222: TEST_ok_to_upgrade_erasure_pool: jq .all_osds_upgraded 2026-02-21T11:38:03.884 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:222: TEST_ok_to_upgrade_erasure_pool: test false = false 2026-02-21T11:38:03.884 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:223: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[7,2],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:03.885 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:223: TEST_ok_to_upgrade_erasure_pool: jq .ok_to_upgrade 2026-02-21T11:38:03.886 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:223: TEST_ok_to_upgrade_erasure_pool: test true = true 2026-02-21T11:38:03.887 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:224: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[7,2],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:03.887 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:224: TEST_ok_to_upgrade_erasure_pool: jq '.osds_ok_to_upgrade | length' 2026-02-21T11:38:03.887 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:224: TEST_ok_to_upgrade_erasure_pool: bc 2026-02-21T11:38:03.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:224: TEST_ok_to_upgrade_erasure_pool: num_osds_upgradable=2 2026-02-21T11:38:03.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:225: TEST_ok_to_upgrade_erasure_pool: test 2 = 2 2026-02-21T11:38:03.890 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:226: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[7,2],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:03.890 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:226: TEST_ok_to_upgrade_erasure_pool: jq '.osds_upgraded | length' 2026-02-21T11:38:03.890 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:226: TEST_ok_to_upgrade_erasure_pool: bc 2026-02-21T11:38:03.892 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:226: TEST_ok_to_upgrade_erasure_pool: num_osds_upgraded=0 2026-02-21T11:38:03.892 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:227: TEST_ok_to_upgrade_erasure_pool: test 0 -eq 0 2026-02-21T11:38:03.892 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:230: TEST_ok_to_upgrade_erasure_pool: ceph osd pool set ec min_size 8 2026-02-21T11:38:04.203 INFO:tasks.workunit.client.0.trial189.stderr:set pool 1 min_size to 8 2026-02-21T11:38:04.224 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:231: TEST_ok_to_upgrade_erasure_pool: sleep 5 2026-02-21T11:38:09.227 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:232: TEST_ok_to_upgrade_erasure_pool: wait_for_clean 2026-02-21T11:38:09.227 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:38:09.228 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:38:09.228 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:38:09.228 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:38:09.228 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:38:09.228 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:38:09.228 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:38:09.228 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:38:09.228 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:38:09.310 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:38:09.310 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:38:09.311 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:38:09.311 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:38:09.311 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:38:09.311 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:38:09.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:38:09.536 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:38:09.536 INFO:tasks.workunit.client.0.trial189.stderr:2 2026-02-21T11:38:09.537 INFO:tasks.workunit.client.0.trial189.stderr:3 2026-02-21T11:38:09.537 INFO:tasks.workunit.client.0.trial189.stderr:4 2026-02-21T11:38:09.537 INFO:tasks.workunit.client.0.trial189.stderr:5 2026-02-21T11:38:09.537 INFO:tasks.workunit.client.0.trial189.stderr:6 2026-02-21T11:38:09.537 INFO:tasks.workunit.client.0.trial189.stderr:7 2026-02-21T11:38:09.537 INFO:tasks.workunit.client.0.trial189.stderr:8 2026-02-21T11:38:09.537 INFO:tasks.workunit.client.0.trial189.stderr:9' 2026-02-21T11:38:09.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:38:09.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:09.537 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:38:09.600 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836496 2026-02-21T11:38:09.601 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836496 2026-02-21T11:38:09.601 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836496' 2026-02-21T11:38:09.601 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:09.601 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:38:09.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=38654705680 2026-02-21T11:38:09.663 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 38654705680 2026-02-21T11:38:09.663 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836496 1-38654705680' 2026-02-21T11:38:09.663 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:09.663 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-21T11:38:09.726 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=60129542159 2026-02-21T11:38:09.726 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 60129542159 2026-02-21T11:38:09.726 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836496 1-38654705680 2-60129542159' 2026-02-21T11:38:09.726 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:09.727 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-21T11:38:09.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=81604378640 2026-02-21T11:38:09.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 81604378640 2026-02-21T11:38:09.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836496 1-38654705680 2-60129542159 3-81604378640' 2026-02-21T11:38:09.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:09.791 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-21T11:38:09.855 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=98784247822 2026-02-21T11:38:09.855 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 98784247822 2026-02-21T11:38:09.855 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836496 1-38654705680 2-60129542159 3-81604378640 4-98784247822' 2026-02-21T11:38:09.855 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:09.855 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-21T11:38:09.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=120259084303 2026-02-21T11:38:09.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 120259084303 2026-02-21T11:38:09.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836496 1-38654705680 2-60129542159 3-81604378640 4-98784247822 5-120259084303' 2026-02-21T11:38:09.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:09.918 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-21T11:38:09.983 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=141733920781 2026-02-21T11:38:09.983 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 141733920781 2026-02-21T11:38:09.983 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836496 1-38654705680 2-60129542159 3-81604378640 4-98784247822 5-120259084303 6-141733920781' 2026-02-21T11:38:09.983 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:09.983 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-21T11:38:10.047 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=163208757262 2026-02-21T11:38:10.047 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 163208757262 2026-02-21T11:38:10.047 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836496 1-38654705680 2-60129542159 3-81604378640 4-98784247822 5-120259084303 6-141733920781 7-163208757262' 2026-02-21T11:38:10.047 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:10.048 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-21T11:38:10.110 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=180388626445 2026-02-21T11:38:10.110 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 180388626445 2026-02-21T11:38:10.110 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836496 1-38654705680 2-60129542159 3-81604378640 4-98784247822 5-120259084303 6-141733920781 7-163208757262 8-180388626445' 2026-02-21T11:38:10.110 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:10.110 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-21T11:38:10.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=197568495628 2026-02-21T11:38:10.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 197568495628 2026-02-21T11:38:10.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836496 1-38654705680 2-60129542159 3-81604378640 4-98784247822 5-120259084303 6-141733920781 7-163208757262 8-180388626445 9-197568495628' 2026-02-21T11:38:10.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:10.176 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836496 2026-02-21T11:38:10.176 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:10.177 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:38:10.177 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836496 2026-02-21T11:38:10.177 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:10.178 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836496 2026-02-21T11:38:10.178 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2026-02-21T11:38:10.178 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836496 2026-02-21T11:38:10.178 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:38:10.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836496 -lt 21474836496 2026-02-21T11:38:10.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:10.407 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-38654705680 2026-02-21T11:38:10.407 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:10.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:38:10.408 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-38654705680 2026-02-21T11:38:10.408 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:10.409 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=38654705680 2026-02-21T11:38:10.409 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 38654705680' 2026-02-21T11:38:10.410 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 38654705680 2026-02-21T11:38:10.410 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:38:10.639 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 38654705680 -lt 38654705680 2026-02-21T11:38:10.640 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:10.640 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 2-60129542159 2026-02-21T11:38:10.640 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:10.641 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=2 2026-02-21T11:38:10.642 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 2-60129542159 2026-02-21T11:38:10.642 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:10.642 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=60129542159 2026-02-21T11:38:10.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.2 seq 60129542159' 2026-02-21T11:38:10.643 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.2 seq 60129542159 2026-02-21T11:38:10.643 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T11:38:10.867 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 60129542159 -lt 60129542159 2026-02-21T11:38:10.867 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:10.868 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 3-81604378640 2026-02-21T11:38:10.868 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:10.868 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=3 2026-02-21T11:38:10.869 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 3-81604378640 2026-02-21T11:38:10.869 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:10.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=81604378640 2026-02-21T11:38:10.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.3 seq 81604378640' 2026-02-21T11:38:10.870 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.3 seq 81604378640 2026-02-21T11:38:10.871 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-21T11:38:11.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 81604378640 -lt 81604378640 2026-02-21T11:38:11.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:11.095 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 4-98784247822 2026-02-21T11:38:11.095 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:11.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=4 2026-02-21T11:38:11.096 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 4-98784247822 2026-02-21T11:38:11.096 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:11.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=98784247822 2026-02-21T11:38:11.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.4 seq 98784247822' 2026-02-21T11:38:11.098 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.4 seq 98784247822 2026-02-21T11:38:11.098 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-21T11:38:11.324 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 98784247822 -lt 98784247822 2026-02-21T11:38:11.325 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:11.325 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 5-120259084303 2026-02-21T11:38:11.325 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:11.326 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=5 2026-02-21T11:38:11.326 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 5-120259084303 2026-02-21T11:38:11.326 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:11.327 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=120259084303 2026-02-21T11:38:11.328 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.5 seq 120259084303' 2026-02-21T11:38:11.328 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.5 seq 120259084303 2026-02-21T11:38:11.328 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-21T11:38:11.552 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 120259084303 -lt 120259084303 2026-02-21T11:38:11.552 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:11.553 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 6-141733920781 2026-02-21T11:38:11.553 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:11.554 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=6 2026-02-21T11:38:11.554 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 6-141733920781 2026-02-21T11:38:11.554 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:11.555 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=141733920781 2026-02-21T11:38:11.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.6 seq 141733920781' 2026-02-21T11:38:11.556 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.6 seq 141733920781 2026-02-21T11:38:11.556 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-21T11:38:11.778 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 141733920781 -lt 141733920781 2026-02-21T11:38:11.779 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:11.779 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 7-163208757262 2026-02-21T11:38:11.779 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:11.780 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=7 2026-02-21T11:38:11.781 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 7-163208757262 2026-02-21T11:38:11.781 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:11.782 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=163208757262 2026-02-21T11:38:11.782 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.7 seq 163208757262' 2026-02-21T11:38:11.782 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.7 seq 163208757262 2026-02-21T11:38:11.782 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-21T11:38:12.006 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 163208757262 -lt 163208757262 2026-02-21T11:38:12.006 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:12.007 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 8-180388626445 2026-02-21T11:38:12.007 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:12.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=8 2026-02-21T11:38:12.008 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 8-180388626445 2026-02-21T11:38:12.008 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:12.009 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=180388626445 2026-02-21T11:38:12.009 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.8 seq 180388626445' 2026-02-21T11:38:12.010 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.8 seq 180388626445 2026-02-21T11:38:12.010 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-21T11:38:12.235 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 180388626446 -lt 180388626445 2026-02-21T11:38:12.235 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:12.235 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 9-197568495628 2026-02-21T11:38:12.236 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:12.236 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=9 2026-02-21T11:38:12.236 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 9-197568495628 2026-02-21T11:38:12.237 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:12.237 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=197568495628 2026-02-21T11:38:12.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.9 seq 197568495628' 2026-02-21T11:38:12.238 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.9 seq 197568495628 2026-02-21T11:38:12.238 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-21T11:38:12.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 197568495628 -lt 197568495628 2026-02-21T11:38:12.458 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:38:12.458 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:38:12.459 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:38:12.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-21T11:38:12.770 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:38:12.770 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:38:12.770 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:38:12.770 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:38:12.770 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:38:12.770 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:38:12.770 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:38:12.994 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-21T11:38:12.995 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:38:12.995 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:38:12.995 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:38:13.308 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-21T11:38:13.308 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:38:13.308 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:38:13.308 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:233: TEST_ok_to_upgrade_erasure_pool: exp_osds_upgradable=0 2026-02-21T11:38:13.308 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:234: TEST_ok_to_upgrade_erasure_pool: ceph osd ok-to-upgrade trial189 01.2.3-1234-g1234deed --format=json 2026-02-21T11:38:13.464 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: unsafe to upgrade osd(s) at this time (1 PGs are or would become offline) 2026-02-21T11:38:13.467 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:234: TEST_ok_to_upgrade_erasure_pool: res=' 2026-02-21T11:38:13.468 INFO:tasks.workunit.client.0.trial189.stderr:{"ok_to_upgrade":false,"all_osds_upgraded":false,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:13.468 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:236: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":false,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:13.468 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:236: TEST_ok_to_upgrade_erasure_pool: jq .all_osds_upgraded 2026-02-21T11:38:13.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:236: TEST_ok_to_upgrade_erasure_pool: test false = false 2026-02-21T11:38:13.471 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:237: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":false,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:13.471 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:237: TEST_ok_to_upgrade_erasure_pool: jq .ok_to_upgrade 2026-02-21T11:38:13.472 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:237: TEST_ok_to_upgrade_erasure_pool: test false = false 2026-02-21T11:38:13.473 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:238: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":false,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:13.473 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:238: TEST_ok_to_upgrade_erasure_pool: jq '.osds_ok_to_upgrade | length' 2026-02-21T11:38:13.473 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:238: TEST_ok_to_upgrade_erasure_pool: bc 2026-02-21T11:38:13.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:238: TEST_ok_to_upgrade_erasure_pool: num_osds_upgradable=0 2026-02-21T11:38:13.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:239: TEST_ok_to_upgrade_erasure_pool: test 0 = 0 2026-02-21T11:38:13.476 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:240: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":false,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:13.476 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:240: TEST_ok_to_upgrade_erasure_pool: jq '.osds_upgraded | length' 2026-02-21T11:38:13.476 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:240: TEST_ok_to_upgrade_erasure_pool: bc 2026-02-21T11:38:13.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:240: TEST_ok_to_upgrade_erasure_pool: num_osds_upgraded=0 2026-02-21T11:38:13.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:241: TEST_ok_to_upgrade_erasure_pool: test 0 -eq 0 2026-02-21T11:38:13.478 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:245: TEST_ok_to_upgrade_erasure_pool: ceph osd metadata 0 --format=json 2026-02-21T11:38:13.479 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:245: TEST_ok_to_upgrade_erasure_pool: jq .ceph_version_short 2026-02-21T11:38:13.479 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:245: TEST_ok_to_upgrade_erasure_pool: sed 's/"//g' 2026-02-21T11:38:13.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:245: TEST_ok_to_upgrade_erasure_pool: ceph_version=20.3.0-5397-gf7273f30 2026-02-21T11:38:13.705 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:246: TEST_ok_to_upgrade_erasure_pool: ceph osd ok-to-upgrade trial189 20.3.0-5397-gf7273f30 --format=json 2026-02-21T11:38:13.919 INFO:tasks.workunit.client.0.trial189.stderr:all 10 osd(s) are running the new Ceph version(20.3.0-5397-gf7273f30) 2026-02-21T11:38:13.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:246: TEST_ok_to_upgrade_erasure_pool: res=' 2026-02-21T11:38:13.927 INFO:tasks.workunit.client.0.trial189.stderr:{"ok_to_upgrade":false,"all_osds_upgraded":true,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[7,2,4,6,5,9,1,3,8,0],"bad_no_version":[]}' 2026-02-21T11:38:13.927 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:247: TEST_ok_to_upgrade_erasure_pool: jq .all_osds_upgraded 2026-02-21T11:38:13.928 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:247: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":true,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[7,2,4,6,5,9,1,3,8,0],"bad_no_version":[]}' 2026-02-21T11:38:13.929 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:247: TEST_ok_to_upgrade_erasure_pool: test true = true 2026-02-21T11:38:13.930 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:248: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":true,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[7,2,4,6,5,9,1,3,8,0],"bad_no_version":[]}' 2026-02-21T11:38:13.930 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:248: TEST_ok_to_upgrade_erasure_pool: jq .ok_to_upgrade 2026-02-21T11:38:13.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:248: TEST_ok_to_upgrade_erasure_pool: test false = false 2026-02-21T11:38:13.933 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:249: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":true,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[7,2,4,6,5,9,1,3,8,0],"bad_no_version":[]}' 2026-02-21T11:38:13.933 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:249: TEST_ok_to_upgrade_erasure_pool: jq '.osds_ok_to_upgrade | length' 2026-02-21T11:38:13.933 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:249: TEST_ok_to_upgrade_erasure_pool: bc 2026-02-21T11:38:13.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:249: TEST_ok_to_upgrade_erasure_pool: num_osds_upgradable=0 2026-02-21T11:38:13.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:250: TEST_ok_to_upgrade_erasure_pool: test 0 -eq 0 2026-02-21T11:38:13.935 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:251: TEST_ok_to_upgrade_erasure_pool: jq '.osds_upgraded | length' 2026-02-21T11:38:13.935 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:251: TEST_ok_to_upgrade_erasure_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":true,"osds_in_crush_bucket":[7,2,4,6,5,9,1,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[7,2,4,6,5,9,1,3,8,0],"bad_no_version":[]}' 2026-02-21T11:38:13.935 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:251: TEST_ok_to_upgrade_erasure_pool: bc 2026-02-21T11:38:13.937 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:251: TEST_ok_to_upgrade_erasure_pool: num_osds_upgraded=10 2026-02-21T11:38:13.937 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:252: TEST_ok_to_upgrade_erasure_pool: test 10 -eq 10 2026-02-21T11:38:13.937 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:33: run: kill_daemons td/ok-to-upgrade KILL 2026-02-21T11:38:13.938 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:38:13.938 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:38:13.938 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:38:13.938 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:38:13.938 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:38:14.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:38:14.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:34: run: teardown td/ok-to-upgrade 2026-02-21T11:38:14.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-upgrade 2026-02-21T11:38:14.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:38:14.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-upgrade KILL 2026-02-21T11:38:14.051 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:38:14.051 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:38:14.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:38:14.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:38:14.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:38:14.057 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:38:14.057 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:38:14.058 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:38:14.058 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:38:14.059 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:38:14.059 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:38:14.059 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:38:14.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:38:14.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:38:14.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:38:14.061 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:38:14.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:38:14.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:38:14.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-upgrade 2026-02-21T11:38:14.089 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:38:14.090 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:14.090 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:14.090 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73624 2026-02-21T11:38:14.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:38:14.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:38:14.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:30: run: for func in $funcs 2026-02-21T11:38:14.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:31: run: setup td/ok-to-upgrade 2026-02-21T11:38:14.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-upgrade 2026-02-21T11:38:14.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-upgrade 2026-02-21T11:38:14.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-upgrade 2026-02-21T11:38:14.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:38:14.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-upgrade KILL 2026-02-21T11:38:14.092 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:38:14.092 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:38:14.093 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:38:14.093 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:38:14.093 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:38:14.093 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:38:14.093 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:38:14.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:38:14.094 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:38:14.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:38:14.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:38:14.095 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:38:14.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:38:14.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:38:14.097 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:38:14.097 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:38:14.097 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:38:14.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:38:14.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-upgrade 2026-02-21T11:38:14.100 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:38:14.100 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:14.100 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:14.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73624 2026-02-21T11:38:14.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:38:14.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:38:14.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-upgrade 2026-02-21T11:38:14.102 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:38:14.102 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:14.102 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:14.102 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73624 2026-02-21T11:38:14.103 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:38:14.104 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:38:14.104 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:38:14.105 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-upgrade 1' TERM HUP INT 2026-02-21T11:38:14.105 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:32: run: TEST_ok_to_upgrade_invalid_args td/ok-to-upgrade 2026-02-21T11:38:14.105 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:39: TEST_ok_to_upgrade_invalid_args: local dir=td/ok-to-upgrade 2026-02-21T11:38:14.105 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:40: TEST_ok_to_upgrade_invalid_args: CEPH_ARGS='--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:38:14.105 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:42: TEST_ok_to_upgrade_invalid_args: run_mon td/ok-to-upgrade a --public-addr=127.0.0.1:7170 2026-02-21T11:38:14.105 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-upgrade 2026-02-21T11:38:14.105 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:38:14.105 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:38:14.105 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:38:14.105 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-upgrade/a 2026-02-21T11:38:14.105 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-upgrade/a --run-dir=td/ok-to-upgrade --public-addr=127.0.0.1:7170 2026-02-21T11:38:14.123 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:38:14.123 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:38:14.123 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:38:14.123 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:38:14.123 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:14.123 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:14.123 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:14.124 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-upgrade/a '--log-file=td/ok-to-upgrade/$name.log' '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-upgrade/log --run-dir=td/ok-to-upgrade '--pid-file=td/ok-to-upgrade/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7170 2026-02-21T11:38:14.147 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:38:14.148 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:38:14.148 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:38:14.148 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:38:14.148 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:38:14.148 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:38:14.148 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:38:14.148 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:38:14.148 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:38:14.149 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:38:14.149 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:14.149 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:14.149 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73624/ceph-mon.a.asok 2026-02-21T11:38:14.149 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:38:14.149 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73624/ceph-mon.a.asok config get fsid 2026-02-21T11:38:14.193 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:38:14.193 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:38:14.193 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:38:14.193 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:38:14.193 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:38:14.194 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:38:14.194 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:38:14.194 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:38:14.195 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:38:14.195 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:14.195 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:14.195 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73624/ceph-mon.a.asok 2026-02-21T11:38:14.195 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:38:14.195 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73624/ceph-mon.a.asok config get mon_host 2026-02-21T11:38:14.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:43: TEST_ok_to_upgrade_invalid_args: run_mgr td/ok-to-upgrade x 2026-02-21T11:38:14.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-upgrade 2026-02-21T11:38:14.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:38:14.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:38:14.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:38:14.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-upgrade/x 2026-02-21T11:38:14.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:38:14.337 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:38:14.337 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:38:14.337 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:38:14.337 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:38:14.338 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:14.338 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:14.338 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:14.338 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:38:14.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-upgrade/x '--log-file=td/ok-to-upgrade/$name.log' '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --run-dir=td/ok-to-upgrade '--pid-file=td/ok-to-upgrade/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:38:14.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:44: TEST_ok_to_upgrade_invalid_args: run_osd td/ok-to-upgrade 0 --osd-mclock-skip-benchmark=true 2026-02-21T11:38:14.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:38:14.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:38:14.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:38:14.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:38:14.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/0 2026-02-21T11:38:14.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:38:14.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:38:14.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:38:14.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:38:14.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/0' 2026-02-21T11:38:14.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/0/journal' 2026-02-21T11:38:14.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:38:14.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:38:14.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:38:14.355 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:38:14.355 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:38:14.356 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:38:14.356 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:38:14.356 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:14.356 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:14.356 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:14.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:14.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:38:14.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:38:14.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:38:14.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:38:14.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:38:14.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:38:14.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:38:14.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:38:14.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:38:14.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:38:14.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:38:14.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/0 2026-02-21T11:38:14.357 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:38:14.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e9a7181c-da9c-4c8a-a78e-a4866518afe8 2026-02-21T11:38:14.358 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e9a7181c-da9c-4c8a-a78e-a4866518afe8' 2026-02-21T11:38:14.358 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 e9a7181c-da9c-4c8a-a78e-a4866518afe8 2026-02-21T11:38:14.358 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:38:14.363 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAmmZlpHvCQFRAAD8jz1mHLjDQXhDyVBqIe0w== 2026-02-21T11:38:14.363 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAmmZlpHvCQFRAAD8jz1mHLjDQXhDyVBqIe0w=="}' 2026-02-21T11:38:14.363 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e9a7181c-da9c-4c8a-a78e-a4866518afe8 -i td/ok-to-upgrade/0/new.json 2026-02-21T11:38:14.460 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:38:14.466 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/0/new.json 2026-02-21T11:38:14.467 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/0 --osd-journal=td/ok-to-upgrade/0/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQAmmZlpHvCQFRAAD8jz1mHLjDQXhDyVBqIe0w== --osd-uuid e9a7181c-da9c-4c8a-a78e-a4866518afe8 2026-02-21T11:38:14.482 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:14.480+0000 7facb3a11780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:14.484 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:14.482+0000 7facb3a11780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:14.484 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:14.483+0000 7facb3a11780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:14.485 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:14.483+0000 7facb3a11780 -1 bdev(0x2d1cc00 td/ok-to-upgrade/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:38:14.485 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:14.483+0000 7facb3a11780 -1 bluestore(td/ok-to-upgrade/0) _read_fsid unparsable uuid 2026-02-21T11:38:14.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/0/keyring 2026-02-21T11:38:14.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:38:14.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:38:14.914 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:38:14.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:38:15.016 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:38:15.016 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:38:15.017 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/0 --osd-journal=td/ok-to-upgrade/0/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:38:15.017 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:38:15.017 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:38:15.017 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:38:15.029 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:15.027+0000 7f394c438780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:15.031 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:15.029+0000 7f394c438780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:15.032 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:15.030+0000 7f394c438780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:15.111 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:38:15.111 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:38:15.112 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:38:15.112 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:38:15.112 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:38:15.112 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:15.112 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:38:15.112 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:38:15.113 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:15.113 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:38:15.206 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:38:15.233 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:15.232+0000 7f394c438780 -1 Falling back to public interface 2026-02-21T11:38:15.535 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:15.533+0000 7f394c438780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:38:16.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:38:16.208 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:16.208 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:38:16.208 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:38:16.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:16.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:38:16.305 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2436366699,v1:127.0.0.1:6801/2436366699] [v2:127.0.0.1:6802/2436366699,v1:127.0.0.1:6803/2436366699] exists,up e9a7181c-da9c-4c8a-a78e-a4866518afe8 2026-02-21T11:38:16.305 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:38:16.306 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:38:16.306 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:38:16.306 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:47: TEST_ok_to_upgrade_invalid_args: ceph osd ok-to-upgrade 2026-02-21T11:38:16.375 INFO:tasks.workunit.client.0.trial189.stderr:Invalid command: missing required parameter crush_bucket() 2026-02-21T11:38:16.375 INFO:tasks.workunit.client.0.trial189.stderr:osd ok-to-upgrade [] : determine a safe number of osd(s) subject to a maximum(if specified) within the provided CRUSH bucket that can be safely upgraded without reducing immediate data availability 2026-02-21T11:38:16.376 INFO:tasks.workunit.client.0.trial189.stderr:Error EINVAL: invalid command 2026-02-21T11:38:16.376 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:50: TEST_ok_to_upgrade_invalid_args: local crush_bucket=foo 2026-02-21T11:38:16.377 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:51: TEST_ok_to_upgrade_invalid_args: local ceph_version_short=01.2.3-1234-g1234deed 2026-02-21T11:38:16.377 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:52: TEST_ok_to_upgrade_invalid_args: ceph osd ok-to-upgrade foo 01.2.3-1234-g1234deed 2026-02-21T11:38:16.456 INFO:tasks.workunit.client.0.trial189.stderr:Error ENOENT: "foo" does not exist 2026-02-21T11:38:16.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:55: TEST_ok_to_upgrade_invalid_args: crush_bucket=default 2026-02-21T11:38:16.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:56: TEST_ok_to_upgrade_invalid_args: ceph osd ok-to-upgrade default 01.2.3-1234-g1234deed 2026-02-21T11:38:16.615 INFO:tasks.workunit.client.0.trial189.stderr:Error EINVAL: crush bucket "default" of type "root" is incompatible for upgradability check; valid types are: 'rack', 'chassis', 'host' and 'osd' 2026-02-21T11:38:16.619 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:59: TEST_ok_to_upgrade_invalid_args: ceph osd tree 2026-02-21T11:38:16.619 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:59: TEST_ok_to_upgrade_invalid_args: grep host 2026-02-21T11:38:16.619 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:59: TEST_ok_to_upgrade_invalid_args: awk '{ print $4 }' 2026-02-21T11:38:16.842 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:59: TEST_ok_to_upgrade_invalid_args: crush_bucket=trial189 2026-02-21T11:38:16.842 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:61: TEST_ok_to_upgrade_invalid_args: ceph_versions=("" "foo" "20" "20.3.0" "20.3.0-1234" "20.3.0-1234-g" "20.1.0-145.el") 2026-02-21T11:38:16.843 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:62: TEST_ok_to_upgrade_invalid_args: for ver in "${ceph_versions[@]}" 2026-02-21T11:38:16.843 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:63: TEST_ok_to_upgrade_invalid_args: ceph osd ok-to-upgrade trial189 2026-02-21T11:38:16.997 INFO:tasks.workunit.client.0.trial189.stderr:Invalid command: missing required parameter ceph_version() 2026-02-21T11:38:16.997 INFO:tasks.workunit.client.0.trial189.stderr:osd ok-to-upgrade [] : determine a safe number of osd(s) subject to a maximum(if specified) within the provided CRUSH bucket that can be safely upgraded without reducing immediate data availability 2026-02-21T11:38:16.997 INFO:tasks.workunit.client.0.trial189.stderr:Error EINVAL: invalid command 2026-02-21T11:38:17.000 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:62: TEST_ok_to_upgrade_invalid_args: for ver in "${ceph_versions[@]}" 2026-02-21T11:38:17.000 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:63: TEST_ok_to_upgrade_invalid_args: ceph osd ok-to-upgrade trial189 foo 2026-02-21T11:38:17.154 INFO:tasks.workunit.client.0.trial189.stderr:Error EINVAL: Invalid Ceph version (short) format. The format to use is the same as 'ceph_version_short' found in OSD metadata. Examples: "20.3.0-3803-g63ca1ffb5a2", "20.1.0-144.el9cp". 2026-02-21T11:38:17.158 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:62: TEST_ok_to_upgrade_invalid_args: for ver in "${ceph_versions[@]}" 2026-02-21T11:38:17.159 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:63: TEST_ok_to_upgrade_invalid_args: ceph osd ok-to-upgrade trial189 20 2026-02-21T11:38:17.316 INFO:tasks.workunit.client.0.trial189.stderr:Error EINVAL: Invalid Ceph version (short) format. The format to use is the same as 'ceph_version_short' found in OSD metadata. Examples: "20.3.0-3803-g63ca1ffb5a2", "20.1.0-144.el9cp". 2026-02-21T11:38:17.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:62: TEST_ok_to_upgrade_invalid_args: for ver in "${ceph_versions[@]}" 2026-02-21T11:38:17.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:63: TEST_ok_to_upgrade_invalid_args: ceph osd ok-to-upgrade trial189 20.3.0 2026-02-21T11:38:17.477 INFO:tasks.workunit.client.0.trial189.stderr:Error EINVAL: Invalid Ceph version (short) format. The format to use is the same as 'ceph_version_short' found in OSD metadata. Examples: "20.3.0-3803-g63ca1ffb5a2", "20.1.0-144.el9cp". 2026-02-21T11:38:17.480 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:62: TEST_ok_to_upgrade_invalid_args: for ver in "${ceph_versions[@]}" 2026-02-21T11:38:17.480 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:63: TEST_ok_to_upgrade_invalid_args: ceph osd ok-to-upgrade trial189 20.3.0-1234 2026-02-21T11:38:17.635 INFO:tasks.workunit.client.0.trial189.stderr:Error EINVAL: Invalid Ceph version (short) format. The format to use is the same as 'ceph_version_short' found in OSD metadata. Examples: "20.3.0-3803-g63ca1ffb5a2", "20.1.0-144.el9cp". 2026-02-21T11:38:17.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:62: TEST_ok_to_upgrade_invalid_args: for ver in "${ceph_versions[@]}" 2026-02-21T11:38:17.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:63: TEST_ok_to_upgrade_invalid_args: ceph osd ok-to-upgrade trial189 20.3.0-1234-g 2026-02-21T11:38:17.793 INFO:tasks.workunit.client.0.trial189.stderr:Error EINVAL: Invalid Ceph version (short) format. The format to use is the same as 'ceph_version_short' found in OSD metadata. Examples: "20.3.0-3803-g63ca1ffb5a2", "20.1.0-144.el9cp". 2026-02-21T11:38:17.796 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:62: TEST_ok_to_upgrade_invalid_args: for ver in "${ceph_versions[@]}" 2026-02-21T11:38:17.796 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:63: TEST_ok_to_upgrade_invalid_args: ceph osd ok-to-upgrade trial189 20.1.0-145.el 2026-02-21T11:38:17.952 INFO:tasks.workunit.client.0.trial189.stderr:Error EINVAL: Invalid Ceph version (short) format. The format to use is the same as 'ceph_version_short' found in OSD metadata. Examples: "20.3.0-3803-g63ca1ffb5a2", "20.1.0-144.el9cp". 2026-02-21T11:38:17.955 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:67: TEST_ok_to_upgrade_invalid_args: max=-20 2026-02-21T11:38:17.956 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:68: TEST_ok_to_upgrade_invalid_args: ceph osd ok-to-upgrade trial189 20.1.0-145.el -20 2026-02-21T11:38:18.111 INFO:tasks.workunit.client.0.trial189.stderr:Error EINVAL: Invalid 'max' value: -20. 'max' must be non-negative. 2026-02-21T11:38:18.114 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:33: run: kill_daemons td/ok-to-upgrade KILL 2026-02-21T11:38:18.115 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:38:18.115 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:38:18.115 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:38:18.115 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:38:18.115 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:38:18.223 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:38:18.223 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:34: run: teardown td/ok-to-upgrade 2026-02-21T11:38:18.223 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-upgrade 2026-02-21T11:38:18.223 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:38:18.223 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-upgrade KILL 2026-02-21T11:38:18.223 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:38:18.223 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:38:18.223 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:38:18.224 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:38:18.224 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:38:18.228 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:38:18.228 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:38:18.229 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:38:18.229 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:38:18.230 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:38:18.230 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:38:18.230 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:38:18.231 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:38:18.231 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:38:18.232 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:38:18.232 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:38:18.232 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:38:18.233 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:38:18.233 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-upgrade 2026-02-21T11:38:18.237 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:38:18.237 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:18.237 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:18.237 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73624 2026-02-21T11:38:18.239 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:38:18.239 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:38:18.239 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:30: run: for func in $funcs 2026-02-21T11:38:18.239 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:31: run: setup td/ok-to-upgrade 2026-02-21T11:38:18.239 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-upgrade 2026-02-21T11:38:18.239 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-upgrade 2026-02-21T11:38:18.239 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-upgrade 2026-02-21T11:38:18.239 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:38:18.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-upgrade KILL 2026-02-21T11:38:18.240 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:38:18.240 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:38:18.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:38:18.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:38:18.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:38:18.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:38:18.241 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:38:18.241 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:38:18.242 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:38:18.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:38:18.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:38:18.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:38:18.244 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:38:18.244 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:38:18.244 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:38:18.244 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:38:18.245 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:38:18.246 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:38:18.246 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-upgrade 2026-02-21T11:38:18.247 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:38:18.247 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:18.247 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:18.247 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73624 2026-02-21T11:38:18.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:38:18.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:38:18.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-upgrade 2026-02-21T11:38:18.249 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:38:18.249 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:18.249 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:18.249 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.73624 2026-02-21T11:38:18.250 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:38:18.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:38:18.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:38:18.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-upgrade 1' TERM HUP INT 2026-02-21T11:38:18.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:32: run: TEST_ok_to_upgrade_replicated_pool td/ok-to-upgrade 2026-02-21T11:38:18.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:72: TEST_ok_to_upgrade_replicated_pool: local dir=td/ok-to-upgrade 2026-02-21T11:38:18.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:73: TEST_ok_to_upgrade_replicated_pool: local poolname=test 2026-02-21T11:38:18.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:74: TEST_ok_to_upgrade_replicated_pool: local OSDS=10 2026-02-21T11:38:18.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:75: TEST_ok_to_upgrade_replicated_pool: local ceph_version=01.2.3-1234-g1234deed 2026-02-21T11:38:18.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:77: TEST_ok_to_upgrade_replicated_pool: CEPH_ARGS='--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:38:18.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:79: TEST_ok_to_upgrade_replicated_pool: run_mon td/ok-to-upgrade a --public-addr=127.0.0.1:7170 2026-02-21T11:38:18.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-upgrade 2026-02-21T11:38:18.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:38:18.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:38:18.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:38:18.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-upgrade/a 2026-02-21T11:38:18.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-upgrade/a --run-dir=td/ok-to-upgrade --public-addr=127.0.0.1:7170 2026-02-21T11:38:18.268 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:38:18.269 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:38:18.269 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:38:18.269 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:38:18.269 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:18.269 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:18.269 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:18.269 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-upgrade/a '--log-file=td/ok-to-upgrade/$name.log' '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-upgrade/log --run-dir=td/ok-to-upgrade '--pid-file=td/ok-to-upgrade/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7170 2026-02-21T11:38:18.292 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:38:18.293 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:38:18.294 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:38:18.294 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:38:18.294 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:38:18.294 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:38:18.294 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:38:18.294 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:38:18.294 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:38:18.294 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:38:18.294 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:18.294 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:18.294 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73624/ceph-mon.a.asok 2026-02-21T11:38:18.295 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:38:18.295 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73624/ceph-mon.a.asok config get fsid 2026-02-21T11:38:18.336 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:38:18.336 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:38:18.336 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:38:18.336 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:38:18.336 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:38:18.336 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:38:18.336 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:38:18.337 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:38:18.337 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:38:18.338 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:18.338 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:18.338 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.73624/ceph-mon.a.asok 2026-02-21T11:38:18.338 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:38:18.338 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.73624/ceph-mon.a.asok config get mon_host 2026-02-21T11:38:18.379 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:80: TEST_ok_to_upgrade_replicated_pool: run_mgr td/ok-to-upgrade x 2026-02-21T11:38:18.379 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-upgrade 2026-02-21T11:38:18.379 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:38:18.379 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:38:18.379 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:38:18.379 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-upgrade/x 2026-02-21T11:38:18.379 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:38:18.475 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:38:18.475 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:38:18.475 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:38:18.475 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:38:18.475 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:18.475 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:18.475 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:18.476 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:38:18.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-upgrade/x '--log-file=td/ok-to-upgrade/$name.log' '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --run-dir=td/ok-to-upgrade '--pid-file=td/ok-to-upgrade/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:38:18.491 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:82: TEST_ok_to_upgrade_replicated_pool: expr 10 - 1 2026-02-21T11:38:18.491 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:82: TEST_ok_to_upgrade_replicated_pool: seq 0 9 2026-02-21T11:38:18.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:82: TEST_ok_to_upgrade_replicated_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:38:18.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:84: TEST_ok_to_upgrade_replicated_pool: run_osd td/ok-to-upgrade 0 --osd-mclock-skip-benchmark=true 2026-02-21T11:38:18.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:38:18.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:38:18.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:38:18.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:38:18.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/0 2026-02-21T11:38:18.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:38:18.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:38:18.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:38:18.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:38:18.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/0' 2026-02-21T11:38:18.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/0/journal' 2026-02-21T11:38:18.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:38:18.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:38:18.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:38:18.497 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:38:18.497 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:38:18.498 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:38:18.498 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:38:18.498 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:18.498 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:18.498 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:18.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:18.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:38:18.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:38:18.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:38:18.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:38:18.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:38:18.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:38:18.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:38:18.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:38:18.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:38:18.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:38:18.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:38:18.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/0 2026-02-21T11:38:18.499 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:38:18.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=19bb88df-76ee-417f-987a-50dc31b476f0 2026-02-21T11:38:18.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 19bb88df-76ee-417f-987a-50dc31b476f0' 2026-02-21T11:38:18.500 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 19bb88df-76ee-417f-987a-50dc31b476f0 2026-02-21T11:38:18.500 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:38:18.504 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAqmZlpVNf4HRAAjLlixvGzeWiQXiq9gDzDMA== 2026-02-21T11:38:18.504 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAqmZlpVNf4HRAAjLlixvGzeWiQXiq9gDzDMA=="}' 2026-02-21T11:38:18.504 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 19bb88df-76ee-417f-987a-50dc31b476f0 -i td/ok-to-upgrade/0/new.json 2026-02-21T11:38:18.603 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:38:18.610 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/0/new.json 2026-02-21T11:38:18.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/0 --osd-journal=td/ok-to-upgrade/0/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQAqmZlpVNf4HRAAjLlixvGzeWiQXiq9gDzDMA== --osd-uuid 19bb88df-76ee-417f-987a-50dc31b476f0 2026-02-21T11:38:18.626 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:18.624+0000 7fb049a30780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:18.627 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:18.626+0000 7fb049a30780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:18.628 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:18.627+0000 7fb049a30780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:18.628 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:18.627+0000 7fb049a30780 -1 bdev(0x3530c00 td/ok-to-upgrade/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:38:18.628 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:18.627+0000 7fb049a30780 -1 bluestore(td/ok-to-upgrade/0) _read_fsid unparsable uuid 2026-02-21T11:38:19.081 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/0/keyring 2026-02-21T11:38:19.081 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:38:19.081 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:38:19.082 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:38:19.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:38:19.182 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:38:19.183 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:38:19.184 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/0 --osd-journal=td/ok-to-upgrade/0/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:38:19.184 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:38:19.184 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:38:19.184 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:38:19.196 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:19.195+0000 7fcc8fe39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:19.198 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:19.196+0000 7fcc8fe39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:19.198 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:19.197+0000 7fcc8fe39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:19.280 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:38:19.280 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:38:19.280 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:38:19.280 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:38:19.280 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:38:19.280 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:19.281 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:38:19.281 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:38:19.281 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:19.282 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:38:19.374 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:38:19.423 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:19.422+0000 7fcc8fe39780 -1 Falling back to public interface 2026-02-21T11:38:19.725 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:19.723+0000 7fcc8fe39780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:38:20.375 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:38:20.375 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:20.376 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:38:20.376 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:38:20.376 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:20.376 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:38:20.470 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2693755815,v1:127.0.0.1:6801/2693755815] [v2:127.0.0.1:6802/2693755815,v1:127.0.0.1:6803/2693755815] exists,up 19bb88df-76ee-417f-987a-50dc31b476f0 2026-02-21T11:38:20.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:38:20.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:38:20.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:38:20.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:82: TEST_ok_to_upgrade_replicated_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:38:20.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:84: TEST_ok_to_upgrade_replicated_pool: run_osd td/ok-to-upgrade 1 --osd-mclock-skip-benchmark=true 2026-02-21T11:38:20.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:38:20.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:38:20.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T11:38:20.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:38:20.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/1 2026-02-21T11:38:20.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:38:20.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:38:20.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:38:20.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:38:20.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/1' 2026-02-21T11:38:20.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/1/journal' 2026-02-21T11:38:20.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:38:20.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:38:20.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:38:20.476 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:38:20.476 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:38:20.476 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:38:20.476 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:38:20.476 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:20.476 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:20.476 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:20.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:20.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:38:20.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:38:20.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:38:20.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:38:20.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:38:20.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:38:20.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:38:20.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:38:20.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:38:20.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:38:20.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:38:20.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/1 2026-02-21T11:38:20.477 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:38:20.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=11c0ded6-1747-4b20-b64e-cc4aa494447c 2026-02-21T11:38:20.477 INFO:tasks.workunit.client.0.trial189.stdout:add osd1 11c0ded6-1747-4b20-b64e-cc4aa494447c 2026-02-21T11:38:20.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 11c0ded6-1747-4b20-b64e-cc4aa494447c' 2026-02-21T11:38:20.478 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:38:20.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAsmZlpMUy5HBAAxz9xrbg258n1nFFfWEQ31Q== 2026-02-21T11:38:20.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAsmZlpMUy5HBAAxz9xrbg258n1nFFfWEQ31Q=="}' 2026-02-21T11:38:20.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 11c0ded6-1747-4b20-b64e-cc4aa494447c -i td/ok-to-upgrade/1/new.json 2026-02-21T11:38:20.584 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:38:20.590 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/1/new.json 2026-02-21T11:38:20.591 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/1 --osd-journal=td/ok-to-upgrade/1/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQAsmZlpMUy5HBAAxz9xrbg258n1nFFfWEQ31Q== --osd-uuid 11c0ded6-1747-4b20-b64e-cc4aa494447c 2026-02-21T11:38:20.605 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:20.604+0000 7fbc44c10780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:20.607 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:20.605+0000 7fbc44c10780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:20.608 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:20.606+0000 7fbc44c10780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:20.608 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:20.606+0000 7fbc44c10780 -1 bdev(0x38a8c00 td/ok-to-upgrade/1/block) open stat got: (1) Operation not permitted 2026-02-21T11:38:20.608 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:20.606+0000 7fbc44c10780 -1 bluestore(td/ok-to-upgrade/1) _read_fsid unparsable uuid 2026-02-21T11:38:20.943 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/1/keyring 2026-02-21T11:38:20.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:38:20.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T11:38:20.944 INFO:tasks.workunit.client.0.trial189.stdout:adding osd1 key to auth repository 2026-02-21T11:38:20.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:38:21.260 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T11:38:21.260 INFO:tasks.workunit.client.0.trial189.stdout:start osd.1 2026-02-21T11:38:21.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/1 --osd-journal=td/ok-to-upgrade/1/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:38:21.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:38:21.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:38:21.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:38:21.273 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:21.271+0000 7fa6d8c2e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:21.274 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:21.272+0000 7fa6d8c2e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:21.275 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:21.273+0000 7fa6d8c2e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:21.427 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:21.424+0000 7fa6d8c2e780 -1 Falling back to public interface 2026-02-21T11:38:21.487 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T11:38:21.487 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:38:21.487 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T11:38:21.487 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:38:21.487 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:38:21.487 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:21.488 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:38:21.488 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:38:21.488 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:21.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:38:21.713 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:38:21.757 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:21.756+0000 7fa6d8c2e780 -1 osd.1 0 log_to_monitors true 2026-02-21T11:38:22.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:38:22.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:22.715 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:38:22.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:38:22.716 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:22.716 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:38:22.941 INFO:tasks.workunit.client.0.trial189.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3415092761,v1:127.0.0.1:6811/3415092761] [v2:127.0.0.1:6812/3415092761,v1:127.0.0.1:6813/3415092761] exists,up 11c0ded6-1747-4b20-b64e-cc4aa494447c 2026-02-21T11:38:22.945 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:38:22.945 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:38:22.945 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:38:22.946 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:82: TEST_ok_to_upgrade_replicated_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:38:22.946 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:84: TEST_ok_to_upgrade_replicated_pool: run_osd td/ok-to-upgrade 2 --osd-mclock-skip-benchmark=true 2026-02-21T11:38:22.946 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:38:22.946 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:38:22.946 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T11:38:22.946 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:38:22.946 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/2 2026-02-21T11:38:22.946 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:38:22.946 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:38:22.946 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:38:22.946 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:38:22.946 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/2' 2026-02-21T11:38:22.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/2/journal' 2026-02-21T11:38:22.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:38:22.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:38:22.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:38:22.947 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:38:22.947 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:38:22.947 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:38:22.947 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:38:22.947 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:22.947 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:22.947 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:22.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:22.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:38:22.948 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:38:22.948 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:38:22.948 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:38:22.948 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:38:22.948 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:38:22.948 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:38:22.948 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:38:22.948 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:38:22.948 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:38:22.948 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:38:22.948 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/2 2026-02-21T11:38:22.949 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:38:22.949 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1db2f761-cc27-4f0f-bc4f-c2f159531600 2026-02-21T11:38:22.949 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 1db2f761-cc27-4f0f-bc4f-c2f159531600' 2026-02-21T11:38:22.949 INFO:tasks.workunit.client.0.trial189.stdout:add osd2 1db2f761-cc27-4f0f-bc4f-c2f159531600 2026-02-21T11:38:22.949 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:38:22.955 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAumZlpHObXOBAA8Zt+nlf4V1lcYfa+xA0dTA== 2026-02-21T11:38:22.955 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAumZlpHObXOBAA8Zt+nlf4V1lcYfa+xA0dTA=="}' 2026-02-21T11:38:22.955 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1db2f761-cc27-4f0f-bc4f-c2f159531600 -i td/ok-to-upgrade/2/new.json 2026-02-21T11:38:23.182 INFO:tasks.workunit.client.0.trial189.stdout:2 2026-02-21T11:38:23.190 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/2/new.json 2026-02-21T11:38:23.191 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/2 --osd-journal=td/ok-to-upgrade/2/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQAumZlpHObXOBAA8Zt+nlf4V1lcYfa+xA0dTA== --osd-uuid 1db2f761-cc27-4f0f-bc4f-c2f159531600 2026-02-21T11:38:23.204 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:23.203+0000 7ff371667780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:23.206 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:23.204+0000 7ff371667780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:23.206 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:23.205+0000 7ff371667780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:23.207 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:23.205+0000 7ff371667780 -1 bdev(0x25dcc00 td/ok-to-upgrade/2/block) open stat got: (1) Operation not permitted 2026-02-21T11:38:23.207 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:23.205+0000 7ff371667780 -1 bluestore(td/ok-to-upgrade/2) _read_fsid unparsable uuid 2026-02-21T11:38:23.522 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/2/keyring 2026-02-21T11:38:23.522 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:38:23.523 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T11:38:23.523 INFO:tasks.workunit.client.0.trial189.stdout:adding osd2 key to auth repository 2026-02-21T11:38:23.524 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:38:23.836 INFO:tasks.workunit.client.0.trial189.stdout:start osd.2 2026-02-21T11:38:23.837 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T11:38:23.837 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/2 --osd-journal=td/ok-to-upgrade/2/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:38:23.837 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:38:23.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:38:23.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:38:23.850 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:23.848+0000 7fbb8265a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:23.851 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:23.849+0000 7fbb8265a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:23.852 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:23.850+0000 7fbb8265a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:23.997 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:23.996+0000 7fbb8265a780 -1 Falling back to public interface 2026-02-21T11:38:24.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T11:38:24.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:38:24.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T11:38:24.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:38:24.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:38:24.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:24.062 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:38:24.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:38:24.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:24.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T11:38:24.284 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:38:24.323 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:24.321+0000 7fbb8265a780 -1 osd.2 0 log_to_monitors true 2026-02-21T11:38:25.285 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:38:25.285 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:25.286 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:38:25.286 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:38:25.286 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:25.286 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T11:38:25.513 INFO:tasks.workunit.client.0.trial189.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2079917621,v1:127.0.0.1:6819/2079917621] [v2:127.0.0.1:6820/2079917621,v1:127.0.0.1:6821/2079917621] exists,up 1db2f761-cc27-4f0f-bc4f-c2f159531600 2026-02-21T11:38:25.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:38:25.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:38:25.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:38:25.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:82: TEST_ok_to_upgrade_replicated_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:38:25.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:84: TEST_ok_to_upgrade_replicated_pool: run_osd td/ok-to-upgrade 3 --osd-mclock-skip-benchmark=true 2026-02-21T11:38:25.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:38:25.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:38:25.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-21T11:38:25.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:38:25.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/3 2026-02-21T11:38:25.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:38:25.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:38:25.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:38:25.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:38:25.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/3' 2026-02-21T11:38:25.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/3/journal' 2026-02-21T11:38:25.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:38:25.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:38:25.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:38:25.519 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:38:25.519 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:38:25.519 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:38:25.519 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:38:25.519 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:25.519 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:25.519 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:25.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:25.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:38:25.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:38:25.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:38:25.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:38:25.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:38:25.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:38:25.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:38:25.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:38:25.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:38:25.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:38:25.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:38:25.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/3 2026-02-21T11:38:25.520 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:38:25.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4044138a-6033-49f9-a2bc-56f5612052c5 2026-02-21T11:38:25.521 INFO:tasks.workunit.client.0.trial189.stdout:add osd3 4044138a-6033-49f9-a2bc-56f5612052c5 2026-02-21T11:38:25.521 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 4044138a-6033-49f9-a2bc-56f5612052c5' 2026-02-21T11:38:25.521 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:38:25.526 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAxmZlplx1MHxAAfkrCU6+uL2emJ9IAQ1S6kQ== 2026-02-21T11:38:25.526 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAxmZlplx1MHxAAfkrCU6+uL2emJ9IAQ1S6kQ=="}' 2026-02-21T11:38:25.526 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4044138a-6033-49f9-a2bc-56f5612052c5 -i td/ok-to-upgrade/3/new.json 2026-02-21T11:38:25.754 INFO:tasks.workunit.client.0.trial189.stdout:3 2026-02-21T11:38:25.761 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/3/new.json 2026-02-21T11:38:25.762 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/3 --osd-journal=td/ok-to-upgrade/3/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQAxmZlplx1MHxAAfkrCU6+uL2emJ9IAQ1S6kQ== --osd-uuid 4044138a-6033-49f9-a2bc-56f5612052c5 2026-02-21T11:38:25.776 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:25.775+0000 7f9eb3e32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:25.777 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:25.776+0000 7f9eb3e32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:25.778 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:25.777+0000 7f9eb3e32780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:25.779 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:25.777+0000 7f9eb3e32780 -1 bdev(0x41a6c00 td/ok-to-upgrade/3/block) open stat got: (1) Operation not permitted 2026-02-21T11:38:25.779 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:25.777+0000 7f9eb3e32780 -1 bluestore(td/ok-to-upgrade/3) _read_fsid unparsable uuid 2026-02-21T11:38:26.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/3/keyring 2026-02-21T11:38:26.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:38:26.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-21T11:38:26.101 INFO:tasks.workunit.client.0.trial189.stdout:adding osd3 key to auth repository 2026-02-21T11:38:26.102 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:38:26.418 INFO:tasks.workunit.client.0.trial189.stdout:start osd.3 2026-02-21T11:38:26.419 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-21T11:38:26.420 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/3 --osd-journal=td/ok-to-upgrade/3/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:38:26.420 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:38:26.420 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:38:26.420 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:38:26.432 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:26.430+0000 7f1243a62780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:26.433 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:26.432+0000 7f1243a62780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:26.434 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:26.433+0000 7f1243a62780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:26.574 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:26.573+0000 7f1243a62780 -1 Falling back to public interface 2026-02-21T11:38:26.642 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-21T11:38:26.642 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:38:26.642 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-21T11:38:26.642 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:38:26.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:38:26.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:26.643 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:38:26.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:38:26.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:26.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-21T11:38:26.854 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:26.852+0000 7f1243a62780 -1 osd.3 0 log_to_monitors true 2026-02-21T11:38:26.865 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:38:27.867 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:38:27.867 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:27.868 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:38:27.869 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:38:27.869 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:27.869 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-21T11:38:28.090 INFO:tasks.workunit.client.0.trial189.stdout:osd.3 up in weight 1 up_from 18 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3842317449,v1:127.0.0.1:6827/3842317449] [v2:127.0.0.1:6828/3842317449,v1:127.0.0.1:6829/3842317449] exists,up 4044138a-6033-49f9-a2bc-56f5612052c5 2026-02-21T11:38:28.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:38:28.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:38:28.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:38:28.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:82: TEST_ok_to_upgrade_replicated_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:38:28.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:84: TEST_ok_to_upgrade_replicated_pool: run_osd td/ok-to-upgrade 4 --osd-mclock-skip-benchmark=true 2026-02-21T11:38:28.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:38:28.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:38:28.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-21T11:38:28.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:38:28.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/4 2026-02-21T11:38:28.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:38:28.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:38:28.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:38:28.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:38:28.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/4' 2026-02-21T11:38:28.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/4/journal' 2026-02-21T11:38:28.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:38:28.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:38:28.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:38:28.096 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:38:28.096 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:38:28.097 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:38:28.097 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:38:28.097 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:28.097 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:28.097 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:28.097 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:28.097 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:38:28.097 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:38:28.097 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:38:28.097 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:38:28.097 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:38:28.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:38:28.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:38:28.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:38:28.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:38:28.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:38:28.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:38:28.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/4 2026-02-21T11:38:28.098 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:38:28.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7f36cae1-f094-43a4-9a4d-fa870a149e96 2026-02-21T11:38:28.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 7f36cae1-f094-43a4-9a4d-fa870a149e96' 2026-02-21T11:38:28.099 INFO:tasks.workunit.client.0.trial189.stdout:add osd4 7f36cae1-f094-43a4-9a4d-fa870a149e96 2026-02-21T11:38:28.099 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:38:28.104 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA0mZlpnz8dBhAAj4mdaNzodC5AabTsqpBelQ== 2026-02-21T11:38:28.104 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA0mZlpnz8dBhAAj4mdaNzodC5AabTsqpBelQ=="}' 2026-02-21T11:38:28.104 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7f36cae1-f094-43a4-9a4d-fa870a149e96 -i td/ok-to-upgrade/4/new.json 2026-02-21T11:38:28.332 INFO:tasks.workunit.client.0.trial189.stdout:4 2026-02-21T11:38:28.340 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/4/new.json 2026-02-21T11:38:28.341 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/4 --osd-journal=td/ok-to-upgrade/4/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQA0mZlpnz8dBhAAj4mdaNzodC5AabTsqpBelQ== --osd-uuid 7f36cae1-f094-43a4-9a4d-fa870a149e96 2026-02-21T11:38:28.356 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:28.355+0000 7f3d931d6780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:28.357 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:28.356+0000 7f3d931d6780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:28.358 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:28.357+0000 7f3d931d6780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:28.359 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:28.357+0000 7f3d931d6780 -1 bdev(0x2a4ac00 td/ok-to-upgrade/4/block) open stat got: (1) Operation not permitted 2026-02-21T11:38:28.359 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:28.357+0000 7f3d931d6780 -1 bluestore(td/ok-to-upgrade/4) _read_fsid unparsable uuid 2026-02-21T11:38:28.683 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/4/keyring 2026-02-21T11:38:28.684 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:38:28.684 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-21T11:38:28.685 INFO:tasks.workunit.client.0.trial189.stdout:adding osd4 key to auth repository 2026-02-21T11:38:28.685 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:38:28.999 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-21T11:38:28.999 INFO:tasks.workunit.client.0.trial189.stdout:start osd.4 2026-02-21T11:38:29.000 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/4 --osd-journal=td/ok-to-upgrade/4/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:38:29.000 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:38:29.000 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:38:29.001 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:38:29.012 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:29.011+0000 7fbcf7c17780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:29.014 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:29.012+0000 7fbcf7c17780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:29.014 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:29.013+0000 7fbcf7c17780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:29.149 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:29.146+0000 7fbcf7c17780 -1 Falling back to public interface 2026-02-21T11:38:29.222 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-21T11:38:29.222 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:38:29.222 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-21T11:38:29.222 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:38:29.222 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:38:29.222 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:29.222 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:38:29.223 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:38:29.223 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:29.224 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-21T11:38:29.444 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:38:29.459 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:29.458+0000 7fbcf7c17780 -1 osd.4 0 log_to_monitors true 2026-02-21T11:38:30.445 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:38:30.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:30.446 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:38:30.447 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:38:30.447 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:30.447 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-21T11:38:30.668 INFO:tasks.workunit.client.0.trial189.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3754854325,v1:127.0.0.1:6835/3754854325] [v2:127.0.0.1:6836/3754854325,v1:127.0.0.1:6837/3754854325] exists,up 7f36cae1-f094-43a4-9a4d-fa870a149e96 2026-02-21T11:38:30.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:38:30.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:38:30.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:38:30.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:82: TEST_ok_to_upgrade_replicated_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:38:30.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:84: TEST_ok_to_upgrade_replicated_pool: run_osd td/ok-to-upgrade 5 --osd-mclock-skip-benchmark=true 2026-02-21T11:38:30.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:38:30.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:38:30.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-21T11:38:30.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:38:30.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/5 2026-02-21T11:38:30.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:38:30.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:38:30.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:38:30.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:38:30.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/5' 2026-02-21T11:38:30.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/5/journal' 2026-02-21T11:38:30.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:38:30.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:38:30.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:38:30.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:38:30.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:38:30.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:38:30.673 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:38:30.673 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:30.673 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:30.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:30.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:30.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:38:30.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:38:30.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:38:30.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:38:30.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:38:30.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:38:30.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:38:30.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:38:30.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:38:30.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:38:30.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:38:30.675 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/5 2026-02-21T11:38:30.675 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:38:30.675 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5ea60e39-3ade-4c54-a2ed-ef90779084a9 2026-02-21T11:38:30.675 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 5ea60e39-3ade-4c54-a2ed-ef90779084a9' 2026-02-21T11:38:30.675 INFO:tasks.workunit.client.0.trial189.stdout:add osd5 5ea60e39-3ade-4c54-a2ed-ef90779084a9 2026-02-21T11:38:30.675 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:38:30.680 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA2mZlpkDt+KBAAt7tMVYvoWP2kkd/OPNHf2Q== 2026-02-21T11:38:30.680 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA2mZlpkDt+KBAAt7tMVYvoWP2kkd/OPNHf2Q=="}' 2026-02-21T11:38:30.681 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5ea60e39-3ade-4c54-a2ed-ef90779084a9 -i td/ok-to-upgrade/5/new.json 2026-02-21T11:38:30.907 INFO:tasks.workunit.client.0.trial189.stdout:5 2026-02-21T11:38:30.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/5/new.json 2026-02-21T11:38:30.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/5 --osd-journal=td/ok-to-upgrade/5/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQA2mZlpkDt+KBAAt7tMVYvoWP2kkd/OPNHf2Q== --osd-uuid 5ea60e39-3ade-4c54-a2ed-ef90779084a9 2026-02-21T11:38:30.930 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:30.928+0000 7f6a1806c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:30.931 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:30.930+0000 7f6a1806c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:30.932 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:30.930+0000 7f6a1806c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:30.933 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:30.931+0000 7f6a1806c780 -1 bdev(0x32fcc00 td/ok-to-upgrade/5/block) open stat got: (1) Operation not permitted 2026-02-21T11:38:30.933 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:30.931+0000 7f6a1806c780 -1 bluestore(td/ok-to-upgrade/5) _read_fsid unparsable uuid 2026-02-21T11:38:31.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/5/keyring 2026-02-21T11:38:31.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:38:31.254 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-21T11:38:31.254 INFO:tasks.workunit.client.0.trial189.stdout:adding osd5 key to auth repository 2026-02-21T11:38:31.255 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:38:31.574 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-21T11:38:31.574 INFO:tasks.workunit.client.0.trial189.stdout:start osd.5 2026-02-21T11:38:31.574 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/5 --osd-journal=td/ok-to-upgrade/5/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:38:31.574 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:38:31.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:38:31.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:38:31.588 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:31.586+0000 7f845260a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:31.589 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:31.588+0000 7f845260a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:31.590 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:31.589+0000 7f845260a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:31.720 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:31.718+0000 7f845260a780 -1 Falling back to public interface 2026-02-21T11:38:31.800 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-21T11:38:31.801 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:38:31.801 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-21T11:38:31.801 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:38:31.801 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:38:31.801 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:31.801 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:38:31.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:38:31.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:31.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-21T11:38:32.027 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:38:32.056 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:32.054+0000 7f845260a780 -1 osd.5 0 log_to_monitors true 2026-02-21T11:38:33.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:38:33.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:33.028 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:38:33.029 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:38:33.029 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:33.029 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-21T11:38:33.255 INFO:tasks.workunit.client.0.trial189.stdout:osd.5 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/610934196,v1:127.0.0.1:6843/610934196] [v2:127.0.0.1:6844/610934196,v1:127.0.0.1:6845/610934196] exists,up 5ea60e39-3ade-4c54-a2ed-ef90779084a9 2026-02-21T11:38:33.259 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:38:33.259 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:38:33.259 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:38:33.259 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:82: TEST_ok_to_upgrade_replicated_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:38:33.259 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:84: TEST_ok_to_upgrade_replicated_pool: run_osd td/ok-to-upgrade 6 --osd-mclock-skip-benchmark=true 2026-02-21T11:38:33.259 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:38:33.259 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:38:33.259 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-21T11:38:33.259 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:38:33.260 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/6 2026-02-21T11:38:33.260 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:38:33.260 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:38:33.260 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:38:33.260 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:38:33.260 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/6' 2026-02-21T11:38:33.260 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/6/journal' 2026-02-21T11:38:33.260 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:38:33.260 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:38:33.260 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:38:33.260 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:38:33.260 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:38:33.260 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:38:33.261 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:38:33.261 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:33.261 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:33.261 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:33.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:33.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:38:33.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:38:33.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:38:33.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:38:33.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:38:33.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:38:33.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:38:33.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:38:33.262 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:38:33.262 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:38:33.262 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:38:33.262 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/6 2026-02-21T11:38:33.262 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:38:33.262 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c720e67e-fc2e-40e8-956b-c70dfb383057 2026-02-21T11:38:33.262 INFO:tasks.workunit.client.0.trial189.stdout:add osd6 c720e67e-fc2e-40e8-956b-c70dfb383057 2026-02-21T11:38:33.263 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 c720e67e-fc2e-40e8-956b-c70dfb383057' 2026-02-21T11:38:33.263 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:38:33.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA5mZlpi+TmDxAAs4DjgWFvIueNFJVA89ZkOA== 2026-02-21T11:38:33.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA5mZlpi+TmDxAAs4DjgWFvIueNFJVA89ZkOA=="}' 2026-02-21T11:38:33.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c720e67e-fc2e-40e8-956b-c70dfb383057 -i td/ok-to-upgrade/6/new.json 2026-02-21T11:38:33.497 INFO:tasks.workunit.client.0.trial189.stdout:6 2026-02-21T11:38:33.504 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/6/new.json 2026-02-21T11:38:33.505 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/6 --osd-journal=td/ok-to-upgrade/6/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQA5mZlpi+TmDxAAs4DjgWFvIueNFJVA89ZkOA== --osd-uuid c720e67e-fc2e-40e8-956b-c70dfb383057 2026-02-21T11:38:33.520 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:33.519+0000 7f343842e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:33.522 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:33.520+0000 7f343842e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:33.523 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:33.521+0000 7f343842e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:33.523 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:33.521+0000 7f343842e780 -1 bdev(0x30d6c00 td/ok-to-upgrade/6/block) open stat got: (1) Operation not permitted 2026-02-21T11:38:33.523 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:33.521+0000 7f343842e780 -1 bluestore(td/ok-to-upgrade/6) _read_fsid unparsable uuid 2026-02-21T11:38:33.861 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/6/keyring 2026-02-21T11:38:33.861 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:38:33.861 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-21T11:38:33.862 INFO:tasks.workunit.client.0.trial189.stdout:adding osd6 key to auth repository 2026-02-21T11:38:33.862 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:38:34.191 INFO:tasks.workunit.client.0.trial189.stdout:start osd.6 2026-02-21T11:38:34.191 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-21T11:38:34.192 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/6 --osd-journal=td/ok-to-upgrade/6/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:38:34.192 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:38:34.192 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:38:34.192 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:38:34.205 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:34.203+0000 7f2265644780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:34.206 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:34.204+0000 7f2265644780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:34.207 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:34.205+0000 7f2265644780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:34.370 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:34.369+0000 7f2265644780 -1 Falling back to public interface 2026-02-21T11:38:34.416 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-21T11:38:34.416 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:38:34.416 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-21T11:38:34.416 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:38:34.416 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:38:34.416 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:34.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:38:34.417 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:38:34.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:34.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-21T11:38:34.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:38:34.737 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:34.735+0000 7f2265644780 -1 osd.6 0 log_to_monitors true 2026-02-21T11:38:35.645 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:38:35.646 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:35.646 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:38:35.647 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:38:35.647 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:35.648 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-21T11:38:35.870 INFO:tasks.workunit.client.0.trial189.stdout:osd.6 up in weight 1 up_from 31 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/679055202,v1:127.0.0.1:6851/679055202] [v2:127.0.0.1:6852/679055202,v1:127.0.0.1:6853/679055202] exists,up c720e67e-fc2e-40e8-956b-c70dfb383057 2026-02-21T11:38:35.873 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:38:35.873 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:38:35.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:38:35.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:82: TEST_ok_to_upgrade_replicated_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:38:35.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:84: TEST_ok_to_upgrade_replicated_pool: run_osd td/ok-to-upgrade 7 --osd-mclock-skip-benchmark=true 2026-02-21T11:38:35.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:38:35.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:38:35.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2026-02-21T11:38:35.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:38:35.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/7 2026-02-21T11:38:35.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:38:35.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:38:35.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:38:35.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:38:35.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/7' 2026-02-21T11:38:35.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/7/journal' 2026-02-21T11:38:35.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:38:35.875 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:38:35.875 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:38:35.875 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:38:35.875 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:38:35.875 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:38:35.875 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:38:35.875 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:35.875 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:35.875 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:35.875 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:35.875 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:38:35.875 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:38:35.875 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:38:35.875 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:38:35.875 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:38:35.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:38:35.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:38:35.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:38:35.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:38:35.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:38:35.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:38:35.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/7 2026-02-21T11:38:35.876 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:38:35.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c20dafc6-b856-4e65-8d2e-8549a2cf0570 2026-02-21T11:38:35.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 c20dafc6-b856-4e65-8d2e-8549a2cf0570' 2026-02-21T11:38:35.876 INFO:tasks.workunit.client.0.trial189.stdout:add osd7 c20dafc6-b856-4e65-8d2e-8549a2cf0570 2026-02-21T11:38:35.877 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:38:35.882 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA7mZlpFTyMNBAAjkTQ2IhbDwtnFtumE1qj/w== 2026-02-21T11:38:35.882 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA7mZlpFTyMNBAAjkTQ2IhbDwtnFtumE1qj/w=="}' 2026-02-21T11:38:35.883 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c20dafc6-b856-4e65-8d2e-8549a2cf0570 -i td/ok-to-upgrade/7/new.json 2026-02-21T11:38:36.111 INFO:tasks.workunit.client.0.trial189.stdout:7 2026-02-21T11:38:36.118 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/7/new.json 2026-02-21T11:38:36.119 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/7 --osd-journal=td/ok-to-upgrade/7/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQA7mZlpFTyMNBAAjkTQ2IhbDwtnFtumE1qj/w== --osd-uuid c20dafc6-b856-4e65-8d2e-8549a2cf0570 2026-02-21T11:38:36.134 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:36.132+0000 7f3e8b810780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:36.135 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:36.134+0000 7f3e8b810780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:36.136 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:36.134+0000 7f3e8b810780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:36.136 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:36.135+0000 7f3e8b810780 -1 bdev(0x24a0c00 td/ok-to-upgrade/7/block) open stat got: (1) Operation not permitted 2026-02-21T11:38:36.137 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:36.135+0000 7f3e8b810780 -1 bluestore(td/ok-to-upgrade/7) _read_fsid unparsable uuid 2026-02-21T11:38:36.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/7/keyring 2026-02-21T11:38:36.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:38:36.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2026-02-21T11:38:36.459 INFO:tasks.workunit.client.0.trial189.stdout:adding osd7 key to auth repository 2026-02-21T11:38:36.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:38:36.779 INFO:tasks.workunit.client.0.trial189.stdout:start osd.7 2026-02-21T11:38:36.779 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2026-02-21T11:38:36.779 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/7 --osd-journal=td/ok-to-upgrade/7/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:38:36.779 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:38:36.779 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:38:36.780 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:38:36.793 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:36.791+0000 7fd4acc0c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:36.794 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:36.793+0000 7fd4acc0c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:36.795 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:36.794+0000 7fd4acc0c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:36.931 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:36.929+0000 7fd4acc0c780 -1 Falling back to public interface 2026-02-21T11:38:37.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2026-02-21T11:38:37.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:38:37.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2026-02-21T11:38:37.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:38:37.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:38:37.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:37.009 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:38:37.009 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:38:37.009 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:37.009 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-21T11:38:37.239 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:38:37.256 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:37.254+0000 7fd4acc0c780 -1 osd.7 0 log_to_monitors true 2026-02-21T11:38:38.241 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:38:38.241 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:38.241 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:38:38.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:38:38.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:38.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-21T11:38:38.464 INFO:tasks.workunit.client.0.trial189.stdout:osd.7 up in weight 1 up_from 36 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/3065925559,v1:127.0.0.1:6859/3065925559] [v2:127.0.0.1:6860/3065925559,v1:127.0.0.1:6861/3065925559] exists,up c20dafc6-b856-4e65-8d2e-8549a2cf0570 2026-02-21T11:38:38.467 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:38:38.467 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:38:38.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:38:38.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:82: TEST_ok_to_upgrade_replicated_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:38:38.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:84: TEST_ok_to_upgrade_replicated_pool: run_osd td/ok-to-upgrade 8 --osd-mclock-skip-benchmark=true 2026-02-21T11:38:38.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:38:38.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:38:38.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2026-02-21T11:38:38.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:38:38.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/8 2026-02-21T11:38:38.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:38:38.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:38:38.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:38:38.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:38:38.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/8' 2026-02-21T11:38:38.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/8/journal' 2026-02-21T11:38:38.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:38:38.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:38:38.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:38:38.469 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:38:38.469 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:38:38.469 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:38:38.469 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:38:38.469 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:38.469 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:38.469 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:38.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:38.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:38:38.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:38:38.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:38:38.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:38:38.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:38:38.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:38:38.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:38:38.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:38:38.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:38:38.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:38:38.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:38:38.471 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/8 2026-02-21T11:38:38.471 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:38:38.471 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b73db86b-f5db-45fa-ae27-a7dac10ff879 2026-02-21T11:38:38.471 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 b73db86b-f5db-45fa-ae27-a7dac10ff879' 2026-02-21T11:38:38.471 INFO:tasks.workunit.client.0.trial189.stdout:add osd8 b73db86b-f5db-45fa-ae27-a7dac10ff879 2026-02-21T11:38:38.471 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:38:38.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA+mZlpqJRKHBAAcJjdOjjjcTfAIwKsxPEg/w== 2026-02-21T11:38:38.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA+mZlpqJRKHBAAcJjdOjjjcTfAIwKsxPEg/w=="}' 2026-02-21T11:38:38.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b73db86b-f5db-45fa-ae27-a7dac10ff879 -i td/ok-to-upgrade/8/new.json 2026-02-21T11:38:38.708 INFO:tasks.workunit.client.0.trial189.stdout:8 2026-02-21T11:38:38.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/8/new.json 2026-02-21T11:38:38.716 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/8 --osd-journal=td/ok-to-upgrade/8/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQA+mZlpqJRKHBAAcJjdOjjjcTfAIwKsxPEg/w== --osd-uuid b73db86b-f5db-45fa-ae27-a7dac10ff879 2026-02-21T11:38:38.731 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:38.729+0000 7f0592213780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:38.732 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:38.731+0000 7f0592213780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:38.733 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:38.731+0000 7f0592213780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:38.734 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:38.732+0000 7f0592213780 -1 bdev(0x3774c00 td/ok-to-upgrade/8/block) open stat got: (1) Operation not permitted 2026-02-21T11:38:38.734 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:38.732+0000 7f0592213780 -1 bluestore(td/ok-to-upgrade/8) _read_fsid unparsable uuid 2026-02-21T11:38:39.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/8/keyring 2026-02-21T11:38:39.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:38:39.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2026-02-21T11:38:39.050 INFO:tasks.workunit.client.0.trial189.stdout:adding osd8 key to auth repository 2026-02-21T11:38:39.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:38:39.376 INFO:tasks.workunit.client.0.trial189.stdout:start osd.8 2026-02-21T11:38:39.377 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2026-02-21T11:38:39.377 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/8 --osd-journal=td/ok-to-upgrade/8/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:38:39.377 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:38:39.377 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:38:39.377 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:38:39.390 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:39.388+0000 7fb9a2a30780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:39.391 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:39.390+0000 7fb9a2a30780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:39.392 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:39.390+0000 7fb9a2a30780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:39.516 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:39.514+0000 7fb9a2a30780 -1 Falling back to public interface 2026-02-21T11:38:39.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2026-02-21T11:38:39.600 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:38:39.600 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2026-02-21T11:38:39.600 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:38:39.600 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:38:39.600 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:39.600 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:38:39.600 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:38:39.601 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:39.601 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-21T11:38:39.802 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:39.800+0000 7fb9a2a30780 -1 osd.8 0 log_to_monitors true 2026-02-21T11:38:39.824 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:38:40.826 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:38:40.826 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:40.827 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:38:40.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:38:40.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:40.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-21T11:38:41.047 INFO:tasks.workunit.client.0.trial189.stdout:osd.8 up in weight 1 up_from 41 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6866/1131360143,v1:127.0.0.1:6867/1131360143] [v2:127.0.0.1:6868/1131360143,v1:127.0.0.1:6869/1131360143] exists,up b73db86b-f5db-45fa-ae27-a7dac10ff879 2026-02-21T11:38:41.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:38:41.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:38:41.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:38:41.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:82: TEST_ok_to_upgrade_replicated_pool: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-21T11:38:41.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:84: TEST_ok_to_upgrade_replicated_pool: run_osd td/ok-to-upgrade 9 --osd-mclock-skip-benchmark=true 2026-02-21T11:38:41.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-upgrade 2026-02-21T11:38:41.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:38:41.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2026-02-21T11:38:41.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:38:41.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-upgrade/9 2026-02-21T11:38:41.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 ' 2026-02-21T11:38:41.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:38:41.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:38:41.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:38:41.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-upgrade/9' 2026-02-21T11:38:41.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-upgrade/9/journal' 2026-02-21T11:38:41.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:38:41.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:38:41.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-upgrade' 2026-02-21T11:38:41.053 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:38:41.053 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:38:41.053 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:38:41.053 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:38:41.053 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:38:41.053 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:38:41.053 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:41.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' 2026-02-21T11:38:41.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:38:41.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:38:41.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:38:41.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-upgrade/$name.log' 2026-02-21T11:38:41.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-upgrade/$name.pid' 2026-02-21T11:38:41.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:38:41.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:38:41.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:38:41.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:38:41.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:38:41.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd-mclock-skip-benchmark=true 2026-02-21T11:38:41.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-upgrade/9 2026-02-21T11:38:41.054 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:38:41.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e7d5bc20-3a7b-4065-95f8-384e54230d0b 2026-02-21T11:38:41.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 e7d5bc20-3a7b-4065-95f8-384e54230d0b' 2026-02-21T11:38:41.055 INFO:tasks.workunit.client.0.trial189.stdout:add osd9 e7d5bc20-3a7b-4065-95f8-384e54230d0b 2026-02-21T11:38:41.055 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:38:41.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBBmZlpfDCNAxAABVed8JOdd2Q7udthvKr9Fw== 2026-02-21T11:38:41.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBBmZlpfDCNAxAABVed8JOdd2Q7udthvKr9Fw=="}' 2026-02-21T11:38:41.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e7d5bc20-3a7b-4065-95f8-384e54230d0b -i td/ok-to-upgrade/9/new.json 2026-02-21T11:38:41.293 INFO:tasks.workunit.client.0.trial189.stdout:9 2026-02-21T11:38:41.300 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-upgrade/9/new.json 2026-02-21T11:38:41.301 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/9 --osd-journal=td/ok-to-upgrade/9/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true --mkfs --key AQBBmZlpfDCNAxAABVed8JOdd2Q7udthvKr9Fw== --osd-uuid e7d5bc20-3a7b-4065-95f8-384e54230d0b 2026-02-21T11:38:41.317 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:41.315+0000 7f5c45c56780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:41.318 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:41.316+0000 7f5c45c56780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:41.319 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:41.317+0000 7f5c45c56780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:41.320 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:41.318+0000 7f5c45c56780 -1 bdev(0x421cc00 td/ok-to-upgrade/9/block) open stat got: (1) Operation not permitted 2026-02-21T11:38:41.320 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:41.318+0000 7f5c45c56780 -1 bluestore(td/ok-to-upgrade/9) _read_fsid unparsable uuid 2026-02-21T11:38:41.668 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-upgrade/9/keyring 2026-02-21T11:38:41.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:38:41.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2026-02-21T11:38:41.670 INFO:tasks.workunit.client.0.trial189.stdout:adding osd9 key to auth repository 2026-02-21T11:38:41.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-upgrade/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:38:41.983 INFO:tasks.workunit.client.0.trial189.stdout:start osd.9 2026-02-21T11:38:41.983 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2026-02-21T11:38:41.983 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=88aee985-8011-4f9d-bc3c-372741f800df --auth-supported=none --mon-host=127.0.0.1:7170 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-upgrade/9 --osd-journal=td/ok-to-upgrade/9/journal --chdir= --run-dir=td/ok-to-upgrade '--admin-socket=/tmp/ceph-asok.73624/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-upgrade/$name.log' '--pid-file=td/ok-to-upgrade/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-mclock-skip-benchmark=true 2026-02-21T11:38:41.983 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:38:41.983 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:38:41.984 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:38:41.996 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:41.995+0000 7f02d062c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:41.998 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:41.996+0000 7f02d062c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:41.999 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:41.997+0000 7f02d062c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:38:42.163 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:42.161+0000 7f02d062c780 -1 Falling back to public interface 2026-02-21T11:38:42.206 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2026-02-21T11:38:42.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:38:42.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-21T11:38:42.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:38:42.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:38:42.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:42.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:38:42.207 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:38:42.208 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:42.208 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-21T11:38:42.430 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:38:42.486 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:38:42.484+0000 7f02d062c780 -1 osd.9 0 log_to_monitors true 2026-02-21T11:38:43.431 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:38:43.431 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:38:43.432 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:38:43.432 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:38:43.433 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:38:43.433 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-21T11:38:43.655 INFO:tasks.workunit.client.0.trial189.stdout:osd.9 up in weight 1 up_from 45 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6874/1035909811,v1:127.0.0.1:6875/1035909811] [v2:127.0.0.1:6876/1035909811,v1:127.0.0.1:6877/1035909811] exists,up e7d5bc20-3a7b-4065-95f8-384e54230d0b 2026-02-21T11:38:43.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:38:43.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:38:43.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:38:43.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:87: TEST_ok_to_upgrade_replicated_pool: create_pool test 32 32 2026-02-21T11:38:43.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create test 32 32 2026-02-21T11:38:43.913 INFO:tasks.workunit.client.0.trial189.stderr:pool 'test' already exists 2026-02-21T11:38:43.922 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:38:44.923 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:88: TEST_ok_to_upgrade_replicated_pool: ceph osd pool set test min_size 1 2026-02-21T11:38:45.229 INFO:tasks.workunit.client.0.trial189.stderr:set pool 1 min_size to 1 2026-02-21T11:38:45.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:89: TEST_ok_to_upgrade_replicated_pool: sleep 5 2026-02-21T11:38:50.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:91: TEST_ok_to_upgrade_replicated_pool: wait_for_clean 2026-02-21T11:38:50.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:38:50.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:38:50.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:38:50.242 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:38:50.243 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:38:50.243 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:38:50.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:38:50.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:38:50.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:38:50.329 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:38:50.329 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:38:50.329 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:38:50.329 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:38:50.329 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:38:50.329 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:38:50.551 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:38:50.551 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:38:50.551 INFO:tasks.workunit.client.0.trial189.stderr:2 2026-02-21T11:38:50.551 INFO:tasks.workunit.client.0.trial189.stderr:3 2026-02-21T11:38:50.552 INFO:tasks.workunit.client.0.trial189.stderr:4 2026-02-21T11:38:50.552 INFO:tasks.workunit.client.0.trial189.stderr:5 2026-02-21T11:38:50.552 INFO:tasks.workunit.client.0.trial189.stderr:6 2026-02-21T11:38:50.552 INFO:tasks.workunit.client.0.trial189.stderr:7 2026-02-21T11:38:50.552 INFO:tasks.workunit.client.0.trial189.stderr:8 2026-02-21T11:38:50.552 INFO:tasks.workunit.client.0.trial189.stderr:9' 2026-02-21T11:38:50.552 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:38:50.553 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:50.553 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:38:50.614 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836489 2026-02-21T11:38:50.614 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836489 2026-02-21T11:38:50.614 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836489' 2026-02-21T11:38:50.615 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:50.615 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:38:50.678 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=38654705672 2026-02-21T11:38:50.678 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 38654705672 2026-02-21T11:38:50.678 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836489 1-38654705672' 2026-02-21T11:38:50.678 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:50.678 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-21T11:38:50.741 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=55834574856 2026-02-21T11:38:50.741 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 55834574856 2026-02-21T11:38:50.741 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574856' 2026-02-21T11:38:50.741 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:50.741 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-21T11:38:50.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=77309411335 2026-02-21T11:38:50.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 77309411335 2026-02-21T11:38:50.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574856 3-77309411335' 2026-02-21T11:38:50.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:50.803 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-21T11:38:50.865 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=94489280519 2026-02-21T11:38:50.865 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 94489280519 2026-02-21T11:38:50.866 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574856 3-77309411335 4-94489280519' 2026-02-21T11:38:50.866 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:50.866 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-21T11:38:50.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=111669149703 2026-02-21T11:38:50.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 111669149703 2026-02-21T11:38:50.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574856 3-77309411335 4-94489280519 5-111669149703' 2026-02-21T11:38:50.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:50.928 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-21T11:38:50.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=133143986183 2026-02-21T11:38:50.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 133143986183 2026-02-21T11:38:50.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574856 3-77309411335 4-94489280519 5-111669149703 6-133143986183' 2026-02-21T11:38:50.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:50.990 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-21T11:38:51.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=154618822661 2026-02-21T11:38:51.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 154618822661 2026-02-21T11:38:51.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574856 3-77309411335 4-94489280519 5-111669149703 6-133143986183 7-154618822661' 2026-02-21T11:38:51.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:51.052 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-21T11:38:51.115 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=176093659142 2026-02-21T11:38:51.115 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 176093659142 2026-02-21T11:38:51.115 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574856 3-77309411335 4-94489280519 5-111669149703 6-133143986183 7-154618822661 8-176093659142' 2026-02-21T11:38:51.115 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:38:51.115 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-21T11:38:51.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=193273528324 2026-02-21T11:38:51.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 193273528324 2026-02-21T11:38:51.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-55834574856 3-77309411335 4-94489280519 5-111669149703 6-133143986183 7-154618822661 8-176093659142 9-193273528324' 2026-02-21T11:38:51.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:51.176 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836489 2026-02-21T11:38:51.176 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:51.177 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:38:51.177 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836489 2026-02-21T11:38:51.177 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:51.178 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836489 2026-02-21T11:38:51.178 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2026-02-21T11:38:51.179 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836489 2026-02-21T11:38:51.179 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:38:51.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836486 -lt 21474836489 2026-02-21T11:38:51.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:38:52.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:38:52.402 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:38:52.627 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836489 -lt 21474836489 2026-02-21T11:38:52.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:52.628 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-38654705672 2026-02-21T11:38:52.628 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:52.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:38:52.629 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-38654705672 2026-02-21T11:38:52.629 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:52.630 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=38654705672 2026-02-21T11:38:52.631 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 38654705672 2026-02-21T11:38:52.631 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2026-02-21T11:38:52.631 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:38:52.854 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 38654705672 -lt 38654705672 2026-02-21T11:38:52.855 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:52.855 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 2-55834574856 2026-02-21T11:38:52.855 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:52.855 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=2 2026-02-21T11:38:52.856 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 2-55834574856 2026-02-21T11:38:52.856 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:52.857 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=55834574856 2026-02-21T11:38:52.857 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.2 seq 55834574856 2026-02-21T11:38:52.858 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.2 seq 55834574856' 2026-02-21T11:38:52.858 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T11:38:53.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 55834574856 -lt 55834574856 2026-02-21T11:38:53.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:53.083 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 3-77309411335 2026-02-21T11:38:53.084 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:53.084 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=3 2026-02-21T11:38:53.085 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 3-77309411335 2026-02-21T11:38:53.085 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:53.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=77309411335 2026-02-21T11:38:53.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.3 seq 77309411335' 2026-02-21T11:38:53.086 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.3 seq 77309411335 2026-02-21T11:38:53.087 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-21T11:38:53.312 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 77309411335 -lt 77309411335 2026-02-21T11:38:53.312 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:53.312 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 4-94489280519 2026-02-21T11:38:53.312 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:53.313 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=4 2026-02-21T11:38:53.313 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 4-94489280519 2026-02-21T11:38:53.313 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:53.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=94489280519 2026-02-21T11:38:53.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.4 seq 94489280519' 2026-02-21T11:38:53.315 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.4 seq 94489280519 2026-02-21T11:38:53.315 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-21T11:38:53.540 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 94489280519 -lt 94489280519 2026-02-21T11:38:53.540 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:53.540 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 5-111669149703 2026-02-21T11:38:53.541 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:53.541 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=5 2026-02-21T11:38:53.542 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 5-111669149703 2026-02-21T11:38:53.542 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:53.543 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=111669149703 2026-02-21T11:38:53.543 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.5 seq 111669149703' 2026-02-21T11:38:53.543 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.5 seq 111669149703 2026-02-21T11:38:53.544 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-21T11:38:53.768 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 111669149703 -lt 111669149703 2026-02-21T11:38:53.768 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:53.768 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 6-133143986183 2026-02-21T11:38:53.768 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:53.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=6 2026-02-21T11:38:53.769 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 6-133143986183 2026-02-21T11:38:53.769 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:53.770 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=133143986183 2026-02-21T11:38:53.771 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.6 seq 133143986183 2026-02-21T11:38:53.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.6 seq 133143986183' 2026-02-21T11:38:53.771 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-21T11:38:53.994 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 133143986183 -lt 133143986183 2026-02-21T11:38:53.994 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:53.995 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 7-154618822661 2026-02-21T11:38:53.995 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:53.995 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=7 2026-02-21T11:38:53.996 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 7-154618822661 2026-02-21T11:38:53.996 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:53.997 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=154618822661 2026-02-21T11:38:53.997 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.7 seq 154618822661' 2026-02-21T11:38:53.997 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.7 seq 154618822661 2026-02-21T11:38:53.998 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-21T11:38:54.223 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 154618822661 -lt 154618822661 2026-02-21T11:38:54.223 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:54.224 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 8-176093659142 2026-02-21T11:38:54.224 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:54.225 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=8 2026-02-21T11:38:54.225 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 8-176093659142 2026-02-21T11:38:54.225 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:54.226 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=176093659142 2026-02-21T11:38:54.226 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.8 seq 176093659142' 2026-02-21T11:38:54.227 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.8 seq 176093659142 2026-02-21T11:38:54.227 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-21T11:38:54.452 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 176093659142 -lt 176093659142 2026-02-21T11:38:54.452 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:38:54.453 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 9-193273528324 2026-02-21T11:38:54.453 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:38:54.453 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=9 2026-02-21T11:38:54.454 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 9-193273528324 2026-02-21T11:38:54.454 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:38:54.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=193273528324 2026-02-21T11:38:54.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.9 seq 193273528324' 2026-02-21T11:38:54.455 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.9 seq 193273528324 2026-02-21T11:38:54.456 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-21T11:38:54.678 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 193273528325 -lt 193273528324 2026-02-21T11:38:54.678 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:38:54.678 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:38:54.678 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:38:54.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-21T11:38:54.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:38:54.990 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:38:54.990 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:38:54.990 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:38:54.990 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:38:54.990 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:38:54.990 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:38:55.212 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-21T11:38:55.213 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:38:55.213 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:38:55.213 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:38:55.527 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-21T11:38:55.527 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:38:55.527 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:38:55.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:94: TEST_ok_to_upgrade_replicated_pool: local exp_osds_upgradable=2 2026-02-21T11:38:55.528 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:95: TEST_ok_to_upgrade_replicated_pool: ceph osd tree 2026-02-21T11:38:55.528 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:95: TEST_ok_to_upgrade_replicated_pool: grep host 2026-02-21T11:38:55.528 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:95: TEST_ok_to_upgrade_replicated_pool: awk '{ print $4 }' 2026-02-21T11:38:55.750 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:95: TEST_ok_to_upgrade_replicated_pool: local crush_bucket=trial189 2026-02-21T11:38:55.751 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:96: TEST_ok_to_upgrade_replicated_pool: ceph osd ok-to-upgrade trial189 01.2.3-1234-g1234deed --format=json 2026-02-21T11:38:55.974 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:96: TEST_ok_to_upgrade_replicated_pool: local 'res= 2026-02-21T11:38:55.974 INFO:tasks.workunit.client.0.trial189.stderr:{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[2,5,6,4,9],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:55.975 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:100: TEST_ok_to_upgrade_replicated_pool: jq .all_osds_upgraded 2026-02-21T11:38:55.975 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:100: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[2,5,6,4,9],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:55.976 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:100: TEST_ok_to_upgrade_replicated_pool: test false = false 2026-02-21T11:38:55.977 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:101: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[2,5,6,4,9],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:55.978 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:101: TEST_ok_to_upgrade_replicated_pool: jq .ok_to_upgrade 2026-02-21T11:38:55.979 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:101: TEST_ok_to_upgrade_replicated_pool: test true = true 2026-02-21T11:38:55.980 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:102: TEST_ok_to_upgrade_replicated_pool: jq '.osds_ok_to_upgrade | length' 2026-02-21T11:38:55.980 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:102: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[2,5,6,4,9],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:55.980 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:102: TEST_ok_to_upgrade_replicated_pool: bc 2026-02-21T11:38:55.982 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:102: TEST_ok_to_upgrade_replicated_pool: local num_osds_upgradable=5 2026-02-21T11:38:55.982 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:103: TEST_ok_to_upgrade_replicated_pool: test 5 -ge 2 2026-02-21T11:38:55.983 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:104: TEST_ok_to_upgrade_replicated_pool: jq '.osds_upgraded | length' 2026-02-21T11:38:55.983 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:104: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[2,5,6,4,9],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:55.983 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:104: TEST_ok_to_upgrade_replicated_pool: bc 2026-02-21T11:38:55.985 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:104: TEST_ok_to_upgrade_replicated_pool: local num_osds_upgraded=0 2026-02-21T11:38:55.985 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:105: TEST_ok_to_upgrade_replicated_pool: test 0 -eq 0 2026-02-21T11:38:55.985 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:110: TEST_ok_to_upgrade_replicated_pool: local max=2 2026-02-21T11:38:55.985 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:111: TEST_ok_to_upgrade_replicated_pool: exp_osds_upgradable=2 2026-02-21T11:38:55.985 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:112: TEST_ok_to_upgrade_replicated_pool: crush_bucket=osd.0 2026-02-21T11:38:55.985 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:113: TEST_ok_to_upgrade_replicated_pool: ceph osd ok-to-upgrade osd.0 01.2.3-1234-g1234deed 2 --format=json 2026-02-21T11:38:56.211 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:113: TEST_ok_to_upgrade_replicated_pool: res=' 2026-02-21T11:38:56.211 INFO:tasks.workunit.client.0.trial189.stderr:{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[0],"osds_ok_to_upgrade":[0,2],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:56.212 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:114: TEST_ok_to_upgrade_replicated_pool: jq .all_osds_upgraded 2026-02-21T11:38:56.212 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:114: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[0],"osds_ok_to_upgrade":[0,2],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:56.213 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:114: TEST_ok_to_upgrade_replicated_pool: test false = false 2026-02-21T11:38:56.214 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:115: TEST_ok_to_upgrade_replicated_pool: jq .ok_to_upgrade 2026-02-21T11:38:56.214 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:115: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[0],"osds_ok_to_upgrade":[0,2],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:56.215 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:115: TEST_ok_to_upgrade_replicated_pool: test true = true 2026-02-21T11:38:56.216 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:116: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[0],"osds_ok_to_upgrade":[0,2],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:56.216 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:116: TEST_ok_to_upgrade_replicated_pool: jq '.osds_ok_to_upgrade | length' 2026-02-21T11:38:56.217 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:116: TEST_ok_to_upgrade_replicated_pool: bc 2026-02-21T11:38:56.218 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:116: TEST_ok_to_upgrade_replicated_pool: num_osds_upgradable=2 2026-02-21T11:38:56.219 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:117: TEST_ok_to_upgrade_replicated_pool: test 2 = 2 2026-02-21T11:38:56.219 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:118: TEST_ok_to_upgrade_replicated_pool: test 2 = 2 2026-02-21T11:38:56.219 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:119: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[0],"osds_ok_to_upgrade":[0,2],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:38:56.219 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:119: TEST_ok_to_upgrade_replicated_pool: jq '.osds_upgraded | length' 2026-02-21T11:38:56.219 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:119: TEST_ok_to_upgrade_replicated_pool: bc 2026-02-21T11:38:56.221 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:119: TEST_ok_to_upgrade_replicated_pool: num_osds_upgraded=0 2026-02-21T11:38:56.221 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:120: TEST_ok_to_upgrade_replicated_pool: test 0 -eq 0 2026-02-21T11:38:56.221 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:123: TEST_ok_to_upgrade_replicated_pool: ceph osd pool set test min_size 2 2026-02-21T11:38:56.528 INFO:tasks.workunit.client.0.trial189.stderr:set pool 1 min_size to 2 2026-02-21T11:38:56.541 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:124: TEST_ok_to_upgrade_replicated_pool: sleep 5 2026-02-21T11:39:01.544 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:125: TEST_ok_to_upgrade_replicated_pool: wait_for_clean 2026-02-21T11:39:01.544 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:39:01.544 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:39:01.544 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:39:01.544 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:39:01.545 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:39:01.545 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:39:01.545 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:39:01.545 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:39:01.545 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:39:01.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:39:01.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:39:01.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:39:01.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:39:01.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:39:01.629 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:39:01.851 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:39:01.851 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:39:01.851 INFO:tasks.workunit.client.0.trial189.stderr:2 2026-02-21T11:39:01.852 INFO:tasks.workunit.client.0.trial189.stderr:3 2026-02-21T11:39:01.852 INFO:tasks.workunit.client.0.trial189.stderr:4 2026-02-21T11:39:01.852 INFO:tasks.workunit.client.0.trial189.stderr:5 2026-02-21T11:39:01.852 INFO:tasks.workunit.client.0.trial189.stderr:6 2026-02-21T11:39:01.852 INFO:tasks.workunit.client.0.trial189.stderr:7 2026-02-21T11:39:01.852 INFO:tasks.workunit.client.0.trial189.stderr:8 2026-02-21T11:39:01.852 INFO:tasks.workunit.client.0.trial189.stderr:9' 2026-02-21T11:39:01.852 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:39:01.852 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:01.852 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:39:01.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836493 2026-02-21T11:39:01.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836493 2026-02-21T11:39:01.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836493' 2026-02-21T11:39:01.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:01.916 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:39:01.979 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=38654705676 2026-02-21T11:39:01.979 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 38654705676 2026-02-21T11:39:01.980 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836493 1-38654705676' 2026-02-21T11:39:01.980 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:01.980 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-21T11:39:02.041 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=55834574860 2026-02-21T11:39:02.041 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 55834574860 2026-02-21T11:39:02.041 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-55834574860' 2026-02-21T11:39:02.041 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:02.041 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-21T11:39:02.106 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=77309411339 2026-02-21T11:39:02.106 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 77309411339 2026-02-21T11:39:02.106 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-55834574860 3-77309411339' 2026-02-21T11:39:02.106 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:02.106 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-21T11:39:02.167 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=94489280523 2026-02-21T11:39:02.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 94489280523 2026-02-21T11:39:02.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-55834574860 3-77309411339 4-94489280523' 2026-02-21T11:39:02.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:02.168 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-21T11:39:02.231 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=111669149707 2026-02-21T11:39:02.231 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 111669149707 2026-02-21T11:39:02.231 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-55834574860 3-77309411339 4-94489280523 5-111669149707' 2026-02-21T11:39:02.232 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:02.232 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-21T11:39:02.294 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=133143986187 2026-02-21T11:39:02.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 133143986187 2026-02-21T11:39:02.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-55834574860 3-77309411339 4-94489280523 5-111669149707 6-133143986187' 2026-02-21T11:39:02.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:02.295 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-21T11:39:02.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=154618822665 2026-02-21T11:39:02.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 154618822665 2026-02-21T11:39:02.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-55834574860 3-77309411339 4-94489280523 5-111669149707 6-133143986187 7-154618822665' 2026-02-21T11:39:02.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:02.359 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-21T11:39:02.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=176093659146 2026-02-21T11:39:02.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 176093659146 2026-02-21T11:39:02.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-55834574860 3-77309411339 4-94489280523 5-111669149707 6-133143986187 7-154618822665 8-176093659146' 2026-02-21T11:39:02.422 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:02.422 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-21T11:39:02.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=193273528328 2026-02-21T11:39:02.484 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 193273528328 2026-02-21T11:39:02.484 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-55834574860 3-77309411339 4-94489280523 5-111669149707 6-133143986187 7-154618822665 8-176093659146 9-193273528328' 2026-02-21T11:39:02.484 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:02.484 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836493 2026-02-21T11:39:02.484 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:02.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:39:02.486 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836493 2026-02-21T11:39:02.486 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:02.487 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836493 2026-02-21T11:39:02.487 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2026-02-21T11:39:02.488 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836493 2026-02-21T11:39:02.488 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:39:02.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836493 -lt 21474836493 2026-02-21T11:39:02.710 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:02.710 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-38654705676 2026-02-21T11:39:02.710 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:02.711 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:39:02.711 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-38654705676 2026-02-21T11:39:02.711 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:02.712 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=38654705676 2026-02-21T11:39:02.713 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2026-02-21T11:39:02.713 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 38654705676 2026-02-21T11:39:02.713 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:39:02.939 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 38654705676 -lt 38654705676 2026-02-21T11:39:02.940 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:02.940 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 2-55834574860 2026-02-21T11:39:02.940 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:02.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=2 2026-02-21T11:39:02.942 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 2-55834574860 2026-02-21T11:39:02.942 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:02.942 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=55834574860 2026-02-21T11:39:02.942 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.2 seq 55834574860' 2026-02-21T11:39:02.943 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.2 seq 55834574860 2026-02-21T11:39:02.943 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T11:39:03.166 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 55834574860 -lt 55834574860 2026-02-21T11:39:03.166 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:03.167 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 3-77309411339 2026-02-21T11:39:03.167 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:03.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=3 2026-02-21T11:39:03.169 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 3-77309411339 2026-02-21T11:39:03.169 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:03.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=77309411339 2026-02-21T11:39:03.170 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.3 seq 77309411339' 2026-02-21T11:39:03.170 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.3 seq 77309411339 2026-02-21T11:39:03.170 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-21T11:39:03.396 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 77309411339 -lt 77309411339 2026-02-21T11:39:03.396 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:03.396 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 4-94489280523 2026-02-21T11:39:03.397 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:03.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=4 2026-02-21T11:39:03.398 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 4-94489280523 2026-02-21T11:39:03.398 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:03.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=94489280523 2026-02-21T11:39:03.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.4 seq 94489280523' 2026-02-21T11:39:03.399 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.4 seq 94489280523 2026-02-21T11:39:03.399 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-21T11:39:03.624 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 94489280523 -lt 94489280523 2026-02-21T11:39:03.624 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:03.625 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 5-111669149707 2026-02-21T11:39:03.625 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:03.625 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=5 2026-02-21T11:39:03.626 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 5-111669149707 2026-02-21T11:39:03.626 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:03.627 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=111669149707 2026-02-21T11:39:03.627 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.5 seq 111669149707' 2026-02-21T11:39:03.628 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.5 seq 111669149707 2026-02-21T11:39:03.628 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-21T11:39:03.851 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 111669149707 -lt 111669149707 2026-02-21T11:39:03.851 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:03.852 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 6-133143986187 2026-02-21T11:39:03.852 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:03.852 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=6 2026-02-21T11:39:03.853 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 6-133143986187 2026-02-21T11:39:03.853 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:03.854 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=133143986187 2026-02-21T11:39:03.854 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.6 seq 133143986187' 2026-02-21T11:39:03.855 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.6 seq 133143986187 2026-02-21T11:39:03.855 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-21T11:39:04.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 133143986187 -lt 133143986187 2026-02-21T11:39:04.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:04.081 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 7-154618822665 2026-02-21T11:39:04.081 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:04.081 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=7 2026-02-21T11:39:04.082 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 7-154618822665 2026-02-21T11:39:04.082 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:04.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=154618822665 2026-02-21T11:39:04.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.7 seq 154618822665' 2026-02-21T11:39:04.083 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.7 seq 154618822665 2026-02-21T11:39:04.083 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-21T11:39:04.307 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 154618822665 -lt 154618822665 2026-02-21T11:39:04.307 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:04.308 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 8-176093659146 2026-02-21T11:39:04.308 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:04.308 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=8 2026-02-21T11:39:04.309 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 8-176093659146 2026-02-21T11:39:04.309 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:04.310 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=176093659146 2026-02-21T11:39:04.310 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.8 seq 176093659146' 2026-02-21T11:39:04.310 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.8 seq 176093659146 2026-02-21T11:39:04.311 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-21T11:39:04.539 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 176093659146 -lt 176093659146 2026-02-21T11:39:04.539 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:04.540 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 9-193273528328 2026-02-21T11:39:04.540 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:04.540 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=9 2026-02-21T11:39:04.541 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 9-193273528328 2026-02-21T11:39:04.541 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:04.542 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=193273528328 2026-02-21T11:39:04.542 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.9 seq 193273528328' 2026-02-21T11:39:04.543 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.9 seq 193273528328 2026-02-21T11:39:04.543 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-21T11:39:04.768 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 193273528329 -lt 193273528328 2026-02-21T11:39:04.769 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:39:04.770 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:39:04.770 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:39:05.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-21T11:39:05.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:39:05.083 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:39:05.083 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:39:05.083 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:39:05.083 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:39:05.083 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:39:05.084 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:39:05.307 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-21T11:39:05.308 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:39:05.308 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:39:05.308 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:39:05.620 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-21T11:39:05.621 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:39:05.621 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:39:05.621 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:126: TEST_ok_to_upgrade_replicated_pool: exp_osds_upgradable=1 2026-02-21T11:39:05.621 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:127: TEST_ok_to_upgrade_replicated_pool: ceph osd tree 2026-02-21T11:39:05.621 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:127: TEST_ok_to_upgrade_replicated_pool: grep host 2026-02-21T11:39:05.621 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:127: TEST_ok_to_upgrade_replicated_pool: awk '{ print $4 }' 2026-02-21T11:39:05.841 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:127: TEST_ok_to_upgrade_replicated_pool: crush_bucket=trial189 2026-02-21T11:39:05.842 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:128: TEST_ok_to_upgrade_replicated_pool: ceph osd ok-to-upgrade trial189 01.2.3-1234-g1234deed --format=json 2026-02-21T11:39:06.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:128: TEST_ok_to_upgrade_replicated_pool: res=' 2026-02-21T11:39:06.071 INFO:tasks.workunit.client.0.trial189.stderr:{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[2,7],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:39:06.072 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:130: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[2,7],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:39:06.072 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:130: TEST_ok_to_upgrade_replicated_pool: jq .all_osds_upgraded 2026-02-21T11:39:06.074 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:130: TEST_ok_to_upgrade_replicated_pool: test false = false 2026-02-21T11:39:06.075 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:131: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[2,7],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:39:06.075 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:131: TEST_ok_to_upgrade_replicated_pool: jq .ok_to_upgrade 2026-02-21T11:39:06.077 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:131: TEST_ok_to_upgrade_replicated_pool: test true = true 2026-02-21T11:39:06.078 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:132: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[2,7],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:39:06.078 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:132: TEST_ok_to_upgrade_replicated_pool: jq '.osds_ok_to_upgrade | length' 2026-02-21T11:39:06.078 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:132: TEST_ok_to_upgrade_replicated_pool: bc 2026-02-21T11:39:06.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:132: TEST_ok_to_upgrade_replicated_pool: num_osds_upgradable=2 2026-02-21T11:39:06.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:133: TEST_ok_to_upgrade_replicated_pool: test 2 -ge 1 2026-02-21T11:39:06.081 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:134: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":true,"all_osds_upgraded":false,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[2,7],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:39:06.081 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:134: TEST_ok_to_upgrade_replicated_pool: jq '.osds_upgraded | length' 2026-02-21T11:39:06.081 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:134: TEST_ok_to_upgrade_replicated_pool: bc 2026-02-21T11:39:06.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:134: TEST_ok_to_upgrade_replicated_pool: num_osds_upgraded=0 2026-02-21T11:39:06.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:135: TEST_ok_to_upgrade_replicated_pool: test 0 -eq 0 2026-02-21T11:39:06.084 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:138: TEST_ok_to_upgrade_replicated_pool: ceph osd pool set test min_size 3 2026-02-21T11:39:06.388 INFO:tasks.workunit.client.0.trial189.stderr:set pool 1 min_size to 3 2026-02-21T11:39:06.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:139: TEST_ok_to_upgrade_replicated_pool: sleep 5 2026-02-21T11:39:11.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:140: TEST_ok_to_upgrade_replicated_pool: wait_for_clean 2026-02-21T11:39:11.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:39:11.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:39:11.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:39:11.402 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:39:11.403 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:39:11.403 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:39:11.403 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:39:11.403 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:39:11.403 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:39:11.487 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:39:11.488 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:39:11.488 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:39:11.488 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:39:11.488 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:39:11.488 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:39:11.712 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:39:11.713 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:39:11.713 INFO:tasks.workunit.client.0.trial189.stderr:2 2026-02-21T11:39:11.713 INFO:tasks.workunit.client.0.trial189.stderr:3 2026-02-21T11:39:11.713 INFO:tasks.workunit.client.0.trial189.stderr:4 2026-02-21T11:39:11.713 INFO:tasks.workunit.client.0.trial189.stderr:5 2026-02-21T11:39:11.713 INFO:tasks.workunit.client.0.trial189.stderr:6 2026-02-21T11:39:11.713 INFO:tasks.workunit.client.0.trial189.stderr:7 2026-02-21T11:39:11.713 INFO:tasks.workunit.client.0.trial189.stderr:8 2026-02-21T11:39:11.713 INFO:tasks.workunit.client.0.trial189.stderr:9' 2026-02-21T11:39:11.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:39:11.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:11.714 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:39:11.774 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836497 2026-02-21T11:39:11.774 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836497 2026-02-21T11:39:11.774 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836497' 2026-02-21T11:39:11.775 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:11.775 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:39:11.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=38654705680 2026-02-21T11:39:11.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 38654705680 2026-02-21T11:39:11.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836497 1-38654705680' 2026-02-21T11:39:11.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:11.838 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-21T11:39:11.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=55834574864 2026-02-21T11:39:11.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 55834574864 2026-02-21T11:39:11.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574864' 2026-02-21T11:39:11.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:11.901 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-21T11:39:11.965 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=77309411343 2026-02-21T11:39:11.966 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 77309411343 2026-02-21T11:39:11.966 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574864 3-77309411343' 2026-02-21T11:39:11.966 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:11.966 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.4 flush_pg_stats 2026-02-21T11:39:12.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=94489280527 2026-02-21T11:39:12.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 94489280527 2026-02-21T11:39:12.029 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574864 3-77309411343 4-94489280527' 2026-02-21T11:39:12.029 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:12.029 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.5 flush_pg_stats 2026-02-21T11:39:12.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=111669149711 2026-02-21T11:39:12.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 111669149711 2026-02-21T11:39:12.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574864 3-77309411343 4-94489280527 5-111669149711' 2026-02-21T11:39:12.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:12.092 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.6 flush_pg_stats 2026-02-21T11:39:12.153 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=133143986191 2026-02-21T11:39:12.153 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 133143986191 2026-02-21T11:39:12.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574864 3-77309411343 4-94489280527 5-111669149711 6-133143986191' 2026-02-21T11:39:12.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:12.154 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.7 flush_pg_stats 2026-02-21T11:39:12.216 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=154618822669 2026-02-21T11:39:12.216 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 154618822669 2026-02-21T11:39:12.216 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574864 3-77309411343 4-94489280527 5-111669149711 6-133143986191 7-154618822669' 2026-02-21T11:39:12.217 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:12.217 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.8 flush_pg_stats 2026-02-21T11:39:12.280 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=176093659150 2026-02-21T11:39:12.281 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 176093659150 2026-02-21T11:39:12.281 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574864 3-77309411343 4-94489280527 5-111669149711 6-133143986191 7-154618822669 8-176093659150' 2026-02-21T11:39:12.281 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:12.281 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.9 flush_pg_stats 2026-02-21T11:39:12.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=193273528332 2026-02-21T11:39:12.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 193273528332 2026-02-21T11:39:12.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836497 1-38654705680 2-55834574864 3-77309411343 4-94489280527 5-111669149711 6-133143986191 7-154618822669 8-176093659150 9-193273528332' 2026-02-21T11:39:12.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:12.343 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836497 2026-02-21T11:39:12.344 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:12.344 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:39:12.345 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836497 2026-02-21T11:39:12.345 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:12.346 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836497 2026-02-21T11:39:12.346 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2026-02-21T11:39:12.346 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836497 2026-02-21T11:39:12.347 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:39:12.571 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836497 -lt 21474836497 2026-02-21T11:39:12.571 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:12.572 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-38654705680 2026-02-21T11:39:12.572 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:12.573 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:39:12.573 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-38654705680 2026-02-21T11:39:12.573 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:12.574 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=38654705680 2026-02-21T11:39:12.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 38654705680' 2026-02-21T11:39:12.575 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 38654705680 2026-02-21T11:39:12.575 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:39:12.800 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 38654705680 -lt 38654705680 2026-02-21T11:39:12.801 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:12.801 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 2-55834574864 2026-02-21T11:39:12.801 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:12.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=2 2026-02-21T11:39:12.802 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 2-55834574864 2026-02-21T11:39:12.802 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:12.803 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=55834574864 2026-02-21T11:39:12.804 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.2 seq 55834574864' 2026-02-21T11:39:12.804 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.2 seq 55834574864 2026-02-21T11:39:12.804 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-21T11:39:13.027 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 55834574864 -lt 55834574864 2026-02-21T11:39:13.027 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:13.027 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 3-77309411343 2026-02-21T11:39:13.027 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:13.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=3 2026-02-21T11:39:13.029 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 3-77309411343 2026-02-21T11:39:13.029 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:13.030 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=77309411343 2026-02-21T11:39:13.030 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.3 seq 77309411343 2026-02-21T11:39:13.030 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.3 seq 77309411343' 2026-02-21T11:39:13.030 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-21T11:39:13.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 77309411343 -lt 77309411343 2026-02-21T11:39:13.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:13.254 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 4-94489280527 2026-02-21T11:39:13.254 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:13.255 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=4 2026-02-21T11:39:13.255 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 4-94489280527 2026-02-21T11:39:13.255 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:13.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=94489280527 2026-02-21T11:39:13.257 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.4 seq 94489280527 2026-02-21T11:39:13.257 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.4 seq 94489280527' 2026-02-21T11:39:13.257 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 4 2026-02-21T11:39:13.482 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 94489280527 -lt 94489280527 2026-02-21T11:39:13.482 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:13.483 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 5-111669149711 2026-02-21T11:39:13.483 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:13.483 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=5 2026-02-21T11:39:13.484 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 5-111669149711 2026-02-21T11:39:13.484 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:13.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=111669149711 2026-02-21T11:39:13.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.5 seq 111669149711' 2026-02-21T11:39:13.485 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.5 seq 111669149711 2026-02-21T11:39:13.486 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 5 2026-02-21T11:39:13.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 111669149711 -lt 111669149711 2026-02-21T11:39:13.710 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:13.710 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 6-133143986191 2026-02-21T11:39:13.710 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:13.711 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=6 2026-02-21T11:39:13.711 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 6-133143986191 2026-02-21T11:39:13.711 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:13.713 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=133143986191 2026-02-21T11:39:13.713 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.6 seq 133143986191' 2026-02-21T11:39:13.713 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.6 seq 133143986191 2026-02-21T11:39:13.713 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 6 2026-02-21T11:39:13.937 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 133143986191 -lt 133143986191 2026-02-21T11:39:13.937 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:13.938 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 7-154618822669 2026-02-21T11:39:13.938 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:13.938 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=7 2026-02-21T11:39:13.939 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 7-154618822669 2026-02-21T11:39:13.939 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:13.940 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=154618822669 2026-02-21T11:39:13.940 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.7 seq 154618822669' 2026-02-21T11:39:13.940 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.7 seq 154618822669 2026-02-21T11:39:13.941 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 7 2026-02-21T11:39:14.167 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 154618822669 -lt 154618822669 2026-02-21T11:39:14.167 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:14.168 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 8-176093659150 2026-02-21T11:39:14.168 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:14.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=8 2026-02-21T11:39:14.169 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 8-176093659150 2026-02-21T11:39:14.169 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:14.170 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=176093659150 2026-02-21T11:39:14.170 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.8 seq 176093659150' 2026-02-21T11:39:14.171 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.8 seq 176093659150 2026-02-21T11:39:14.171 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 8 2026-02-21T11:39:14.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 176093659150 -lt 176093659150 2026-02-21T11:39:14.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:14.397 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 9-193273528332 2026-02-21T11:39:14.398 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:14.398 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=9 2026-02-21T11:39:14.399 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 9-193273528332 2026-02-21T11:39:14.399 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:14.400 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=193273528332 2026-02-21T11:39:14.400 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.9 seq 193273528332 2026-02-21T11:39:14.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.9 seq 193273528332' 2026-02-21T11:39:14.401 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 9 2026-02-21T11:39:14.626 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 193273528333 -lt 193273528332 2026-02-21T11:39:14.626 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:39:14.627 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:39:14.627 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:39:14.938 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-21T11:39:14.939 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:39:14.939 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:39:14.939 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:39:14.939 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:39:14.939 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:39:14.939 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:39:14.939 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:39:15.159 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-21T11:39:15.159 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:39:15.159 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:39:15.159 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:39:15.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-21T11:39:15.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:39:15.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:39:15.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:141: TEST_ok_to_upgrade_replicated_pool: exp_osds_upgradable=0 2026-02-21T11:39:15.474 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:142: TEST_ok_to_upgrade_replicated_pool: ceph osd ok-to-upgrade trial189 01.2.3-1234-g1234deed --format=json 2026-02-21T11:39:15.630 INFO:tasks.workunit.client.0.trial189.stderr:Error EBUSY: unsafe to upgrade osd(s) at this time (4 PGs are or would become offline) 2026-02-21T11:39:15.633 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:142: TEST_ok_to_upgrade_replicated_pool: res=' 2026-02-21T11:39:15.633 INFO:tasks.workunit.client.0.trial189.stderr:{"ok_to_upgrade":false,"all_osds_upgraded":false,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:39:15.634 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:144: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":false,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:39:15.634 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:144: TEST_ok_to_upgrade_replicated_pool: jq .all_osds_upgraded 2026-02-21T11:39:15.635 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:144: TEST_ok_to_upgrade_replicated_pool: test false = false 2026-02-21T11:39:15.636 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:145: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":false,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:39:15.636 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:145: TEST_ok_to_upgrade_replicated_pool: jq .ok_to_upgrade 2026-02-21T11:39:15.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:145: TEST_ok_to_upgrade_replicated_pool: test false = false 2026-02-21T11:39:15.639 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:146: TEST_ok_to_upgrade_replicated_pool: jq '.osds_ok_to_upgrade | length' 2026-02-21T11:39:15.639 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:146: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":false,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:39:15.639 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:146: TEST_ok_to_upgrade_replicated_pool: bc 2026-02-21T11:39:15.641 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:146: TEST_ok_to_upgrade_replicated_pool: num_osds_upgradable=0 2026-02-21T11:39:15.641 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:147: TEST_ok_to_upgrade_replicated_pool: test 0 = 0 2026-02-21T11:39:15.641 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:148: TEST_ok_to_upgrade_replicated_pool: jq '.osds_upgraded | length' 2026-02-21T11:39:15.641 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:148: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":false,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[],"bad_no_version":[]}' 2026-02-21T11:39:15.641 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:148: TEST_ok_to_upgrade_replicated_pool: bc 2026-02-21T11:39:15.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:148: TEST_ok_to_upgrade_replicated_pool: num_osds_upgraded=0 2026-02-21T11:39:15.643 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:149: TEST_ok_to_upgrade_replicated_pool: test 0 -eq 0 2026-02-21T11:39:15.644 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:153: TEST_ok_to_upgrade_replicated_pool: ceph osd metadata 0 --format=json 2026-02-21T11:39:15.644 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:153: TEST_ok_to_upgrade_replicated_pool: jq .ceph_version_short 2026-02-21T11:39:15.644 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:153: TEST_ok_to_upgrade_replicated_pool: sed 's/"//g' 2026-02-21T11:39:15.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:153: TEST_ok_to_upgrade_replicated_pool: upgrade_version=20.3.0-5397-gf7273f30 2026-02-21T11:39:15.877 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:154: TEST_ok_to_upgrade_replicated_pool: ceph osd ok-to-upgrade trial189 20.3.0-5397-gf7273f30 --format=json 2026-02-21T11:39:16.097 INFO:tasks.workunit.client.0.trial189.stderr:all 10 osd(s) are running the new Ceph version(20.3.0-5397-gf7273f30) 2026-02-21T11:39:16.105 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:154: TEST_ok_to_upgrade_replicated_pool: res=' 2026-02-21T11:39:16.105 INFO:tasks.workunit.client.0.trial189.stderr:{"ok_to_upgrade":false,"all_osds_upgraded":true,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[2,5,6,7,1,4,9,3,8,0],"bad_no_version":[]}' 2026-02-21T11:39:16.106 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:155: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":true,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[2,5,6,7,1,4,9,3,8,0],"bad_no_version":[]}' 2026-02-21T11:39:16.106 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:155: TEST_ok_to_upgrade_replicated_pool: jq .all_osds_upgraded 2026-02-21T11:39:16.108 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:155: TEST_ok_to_upgrade_replicated_pool: test true = true 2026-02-21T11:39:16.109 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:156: TEST_ok_to_upgrade_replicated_pool: jq .ok_to_upgrade 2026-02-21T11:39:16.109 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:156: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":true,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[2,5,6,7,1,4,9,3,8,0],"bad_no_version":[]}' 2026-02-21T11:39:16.111 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:156: TEST_ok_to_upgrade_replicated_pool: test false = false 2026-02-21T11:39:16.112 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:157: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":true,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[2,5,6,7,1,4,9,3,8,0],"bad_no_version":[]}' 2026-02-21T11:39:16.112 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:157: TEST_ok_to_upgrade_replicated_pool: jq '.osds_ok_to_upgrade | length' 2026-02-21T11:39:16.112 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:157: TEST_ok_to_upgrade_replicated_pool: bc 2026-02-21T11:39:16.114 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:157: TEST_ok_to_upgrade_replicated_pool: num_osds_upgradable=0 2026-02-21T11:39:16.114 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:158: TEST_ok_to_upgrade_replicated_pool: test 0 -eq 0 2026-02-21T11:39:16.114 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:159: TEST_ok_to_upgrade_replicated_pool: echo '{"ok_to_upgrade":false,"all_osds_upgraded":true,"osds_in_crush_bucket":[2,5,6,7,1,4,9,3,8,0],"osds_ok_to_upgrade":[],"osds_upgraded":[2,5,6,7,1,4,9,3,8,0],"bad_no_version":[]}' 2026-02-21T11:39:16.114 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:159: TEST_ok_to_upgrade_replicated_pool: jq '.osds_upgraded | length' 2026-02-21T11:39:16.115 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:159: TEST_ok_to_upgrade_replicated_pool: bc 2026-02-21T11:39:16.116 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:159: TEST_ok_to_upgrade_replicated_pool: num_osds_upgraded=10 2026-02-21T11:39:16.116 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:160: TEST_ok_to_upgrade_replicated_pool: test 10 -eq 10 2026-02-21T11:39:16.116 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:33: run: kill_daemons td/ok-to-upgrade KILL 2026-02-21T11:39:16.117 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:16.117 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:16.117 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:16.118 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:16.118 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:16.228 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:16.228 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-upgrade.sh:34: run: teardown td/ok-to-upgrade 2026-02-21T11:39:16.228 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-upgrade 2026-02-21T11:39:16.228 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:16.229 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-upgrade KILL 2026-02-21T11:39:16.229 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:16.229 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:16.229 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:16.229 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:16.229 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:16.235 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:16.235 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:16.236 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:16.236 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:16.237 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:16.237 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:16.237 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:16.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:16.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:16.239 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:16.239 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:16.239 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:16.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:16.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-upgrade 2026-02-21T11:39:16.261 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:16.261 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:16.261 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:39:16.261 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73624 2026-02-21T11:39:16.263 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:16.263 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:16.263 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: main: code=0 2026-02-21T11:39:16.263 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2433: main: teardown td/ok-to-upgrade 0 2026-02-21T11:39:16.263 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-upgrade 2026-02-21T11:39:16.263 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-21T11:39:16.263 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-upgrade KILL 2026-02-21T11:39:16.263 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:16.263 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:16.263 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:16.264 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:16.264 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:16.264 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:16.265 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:16.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:16.266 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:16.266 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:16.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:16.267 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:16.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:16.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:16.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:16.268 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:16.269 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:16.270 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-21T11:39:16.270 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-upgrade 2026-02-21T11:39:16.271 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:16.271 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:16.271 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.73624 2026-02-21T11:39:16.272 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.73624 2026-02-21T11:39:16.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:16.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:16.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: main: return 0 2026-02-21T11:39:16.273 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-21T11:39:16.274 DEBUG:teuthology.orchestra.run.trial189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-21T11:39:16.337 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2026-02-21T11:39:16.337 DEBUG:teuthology.orchestra.run.trial189:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2026-02-21T11:39:16.400 INFO:tasks.workunit.client.0.trial189.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-21T11:39:16.406 INFO:tasks.workunit.client.0.trial189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-21T11:39:16.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-21T11:39:16.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-21T11:39:16.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: main: export PYTHONWARNINGS=ignore 2026-02-21T11:39:16.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: main: PYTHONWARNINGS=ignore 2026-02-21T11:39:16.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: main: export CEPH_CONF=/dev/null 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: main: CEPH_CONF=/dev/null 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: main: unset CEPH_ARGS 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2427: main: local code 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: main: run td/rados-striper 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=f606df79-e3dc-43f9-ab2a-a1d2bb965573 --auth-supported=none ' 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/rados-striper 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/rados-striper 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:16.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:16.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:16.408 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:16.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:16.408 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:16.409 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:16.409 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:16.409 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:16.410 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:16.410 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:16.411 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:16.411 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:16.411 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:16.412 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:16.412 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2026-02-21T11:39:16.413 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:16.413 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:16.413 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.113465 2026-02-21T11:39:16.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.113465 2026-02-21T11:39:16.414 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:16.414 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:16.414 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/rados-striper 2026-02-21T11:39:16.415 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:39:16.415 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:16.415 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.113465 2026-02-21T11:39:16.415 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.113465 2026-02-21T11:39:16.416 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:39:16.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-21T11:39:16.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-21T11:39:16.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:39:16.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2026-02-21T11:39:16.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2026-02-21T11:39:16.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/rados-striper 2026-02-21T11:39:16.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:39:16.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:39:16.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:39:16.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/rados-striper/a 2026-02-21T11:39:16.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2026-02-21T11:39:16.434 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:39:16.434 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:16.434 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:16.435 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:16.435 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:16.435 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.113465 2026-02-21T11:39:16.435 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.113465/$cluster-$name.asok' 2026-02-21T11:39:16.435 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.113465/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T11:39:16.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:39:16.459 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:39:16.459 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:16.459 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:16.459 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:39:16.459 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:39:16.460 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:16.460 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:16.460 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:16.461 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:16.461 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:16.461 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.113465 2026-02-21T11:39:16.461 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.113465/ceph-mon.a.asok 2026-02-21T11:39:16.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:16.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.113465/ceph-mon.a.asok config get fsid 2026-02-21T11:39:16.504 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:39:16.504 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:16.504 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:16.504 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:39:16.504 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:39:16.505 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:16.505 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:16.505 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:16.505 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:16.505 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:16.505 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.113465 2026-02-21T11:39:16.505 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.113465/ceph-mon.a.asok 2026-02-21T11:39:16.506 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:16.506 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.113465/ceph-mon.a.asok config get mon_host 2026-02-21T11:39:16.547 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2026-02-21T11:39:16.547 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2026-02-21T11:39:16.547 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:39:16.547 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:39:16.547 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/0 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f606df79-e3dc-43f9-ab2a-a1d2bb965573 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.113465 2026-02-21T11:39:16.548 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.113465/$cluster-$name.asok' 2026-02-21T11:39:16.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.113465/$cluster-$name.asok' 2026-02-21T11:39:16.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:39:16.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:39:16.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:39:16.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2026-02-21T11:39:16.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2026-02-21T11:39:16.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:39:16.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:39:16.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:39:16.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:39:16.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:39:16.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:39:16.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/0 2026-02-21T11:39:16.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:39:16.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ed10de61-5c5e-4d09-ba58-d25d075c56f5 2026-02-21T11:39:16.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ed10de61-5c5e-4d09-ba58-d25d075c56f5' 2026-02-21T11:39:16.550 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 ed10de61-5c5e-4d09-ba58-d25d075c56f5 2026-02-21T11:39:16.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:39:16.558 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBkmZlpDw05IRAA+83CrdoyBaBdxobszuo6Fw== 2026-02-21T11:39:16.558 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBkmZlpDw05IRAA+83CrdoyBaBdxobszuo6Fw=="}' 2026-02-21T11:39:16.558 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ed10de61-5c5e-4d09-ba58-d25d075c56f5 -i td/rados-striper/0/new.json 2026-02-21T11:39:16.659 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:16.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/0/new.json 2026-02-21T11:39:16.666 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f606df79-e3dc-43f9-ab2a-a1d2bb965573 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.113465/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBkmZlpDw05IRAA+83CrdoyBaBdxobszuo6Fw== --osd-uuid ed10de61-5c5e-4d09-ba58-d25d075c56f5 2026-02-21T11:39:16.680 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:16.679+0000 7f6d4a044780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:16.681 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:16.681+0000 7f6d4a044780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:16.682 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:16.681+0000 7f6d4a044780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:16.683 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:16.682+0000 7f6d4a044780 -1 bdev(0x2bf2c00 td/rados-striper/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:39:16.683 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:16.682+0000 7f6d4a044780 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2026-02-21T11:39:17.009 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/0/keyring 2026-02-21T11:39:17.009 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:39:17.009 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:39:17.010 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:39:17.010 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:39:17.111 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:39:17.112 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:39:17.112 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f606df79-e3dc-43f9-ab2a-a1d2bb965573 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.113465/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:39:17.112 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:39:17.112 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:39:17.112 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:39:17.124 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:17.123+0000 7f03f6c2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:17.125 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:17.125+0000 7f03f6c2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:17.126 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:17.126+0000 7f03f6c2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:17.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:39:17.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:39:17.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:39:17.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:39:17.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:39:17.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:17.208 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:39:17.208 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:17.208 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:17.208 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:17.300 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:39:17.341 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:17.340+0000 7f03f6c2c780 -1 Falling back to public interface 2026-02-21T11:39:17.595 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:17.594+0000 7f03f6c2c780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:39:18.302 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:39:18.302 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:18.302 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:39:18.302 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:39:18.303 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:18.303 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:18.400 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3249222994,v1:127.0.0.1:6801/3249222994] [v2:127.0.0.1:6802/3249222994,v1:127.0.0.1:6803/3249222994] exists,up ed10de61-5c5e-4d09-ba58-d25d075c56f5 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/1 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f606df79-e3dc-43f9-ab2a-a1d2bb965573 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:18.401 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.113465 2026-02-21T11:39:18.402 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.113465/$cluster-$name.asok' 2026-02-21T11:39:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.113465/$cluster-$name.asok' 2026-02-21T11:39:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:39:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:39:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:39:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2026-02-21T11:39:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2026-02-21T11:39:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:39:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:39:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:39:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:39:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:39:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:39:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/1 2026-02-21T11:39:18.402 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:39:18.403 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d26a580a-ec51-45c3-abaa-4f9d0a2124d1 2026-02-21T11:39:18.403 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d26a580a-ec51-45c3-abaa-4f9d0a2124d1' 2026-02-21T11:39:18.403 INFO:tasks.workunit.client.0.trial189.stdout:add osd1 d26a580a-ec51-45c3-abaa-4f9d0a2124d1 2026-02-21T11:39:18.403 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:39:18.412 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBmmZlp+OSIGBAA2OajmVWG50rJgNHigHNthg== 2026-02-21T11:39:18.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBmmZlp+OSIGBAA2OajmVWG50rJgNHigHNthg=="}' 2026-02-21T11:39:18.413 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d26a580a-ec51-45c3-abaa-4f9d0a2124d1 -i td/rados-striper/1/new.json 2026-02-21T11:39:18.513 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:39:18.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/1/new.json 2026-02-21T11:39:18.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=f606df79-e3dc-43f9-ab2a-a1d2bb965573 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.113465/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBmmZlp+OSIGBAA2OajmVWG50rJgNHigHNthg== --osd-uuid d26a580a-ec51-45c3-abaa-4f9d0a2124d1 2026-02-21T11:39:18.533 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:18.532+0000 7f8377433780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:18.534 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:18.533+0000 7f8377433780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:18.535 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:18.534+0000 7f8377433780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:18.536 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:18.535+0000 7f8377433780 -1 bdev(0x3f7ac00 td/rados-striper/1/block) open stat got: (1) Operation not permitted 2026-02-21T11:39:18.536 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:18.535+0000 7f8377433780 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2026-02-21T11:39:18.854 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/1/keyring 2026-02-21T11:39:18.854 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:39:18.855 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T11:39:18.855 INFO:tasks.workunit.client.0.trial189.stdout:adding osd1 key to auth repository 2026-02-21T11:39:18.856 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:39:18.954 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T11:39:18.954 INFO:tasks.workunit.client.0.trial189.stdout:start osd.1 2026-02-21T11:39:18.954 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=f606df79-e3dc-43f9-ab2a-a1d2bb965573 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.113465/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:39:18.954 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:39:18.954 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:39:18.954 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:39:18.966 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:18.966+0000 7ff4e842b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:18.967 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:18.967+0000 7ff4e842b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:18.968 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:18.968+0000 7ff4e842b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:19.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T11:39:19.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:39:19.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T11:39:19.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:39:19.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:39:19.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:19.049 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:19.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:39:19.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:19.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:39:19.141 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:39:19.160 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:19.159+0000 7ff4e842b780 -1 Falling back to public interface 2026-02-21T11:39:19.406 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:19.405+0000 7ff4e842b780 -1 osd.1 0 log_to_monitors true 2026-02-21T11:39:20.142 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:39:20.143 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:20.143 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:39:20.144 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:39:20.144 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:20.144 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:39:20.237 INFO:tasks.workunit.client.0.trial189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/2488966744,v1:127.0.0.1:6809/2488966744] [v2:127.0.0.1:6810/2488966744,v1:127.0.0.1:6811/2488966744] exists,up d26a580a-ec51-45c3-abaa-4f9d0a2124d1 2026-02-21T11:39:20.241 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:39:20.241 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:39:20.241 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:39:20.241 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2026-02-21T11:39:20.241 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2026-02-21T11:39:20.241 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:39:20.241 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-21T11:39:20.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:39:20.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/2 2026-02-21T11:39:20.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f606df79-e3dc-43f9-ab2a-a1d2bb965573 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2026-02-21T11:39:20.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:39:20.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:39:20.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:39:20.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2026-02-21T11:39:20.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2026-02-21T11:39:20.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:39:20.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.113465 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.113465/$cluster-$name.asok' 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.113465/$cluster-$name.asok' 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:39:20.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:39:20.244 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/2 2026-02-21T11:39:20.244 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:39:20.244 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=93b1e47d-0baf-437b-b434-beed4bc83c9c 2026-02-21T11:39:20.244 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 93b1e47d-0baf-437b-b434-beed4bc83c9c' 2026-02-21T11:39:20.244 INFO:tasks.workunit.client.0.trial189.stdout:add osd2 93b1e47d-0baf-437b-b434-beed4bc83c9c 2026-02-21T11:39:20.244 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:39:20.250 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBomZlpc1vYDhAA+ld+tRqTn/hTNJHpM9zWUw== 2026-02-21T11:39:20.250 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBomZlpc1vYDhAA+ld+tRqTn/hTNJHpM9zWUw=="}' 2026-02-21T11:39:20.250 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 93b1e47d-0baf-437b-b434-beed4bc83c9c -i td/rados-striper/2/new.json 2026-02-21T11:39:20.350 INFO:tasks.workunit.client.0.trial189.stdout:2 2026-02-21T11:39:20.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/2/new.json 2026-02-21T11:39:20.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=f606df79-e3dc-43f9-ab2a-a1d2bb965573 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.113465/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBomZlpc1vYDhAA+ld+tRqTn/hTNJHpM9zWUw== --osd-uuid 93b1e47d-0baf-437b-b434-beed4bc83c9c 2026-02-21T11:39:20.371 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:20.370+0000 7f0a9fe44780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:20.372 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:20.372+0000 7f0a9fe44780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:20.373 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:20.372+0000 7f0a9fe44780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:20.374 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:20.373+0000 7f0a9fe44780 -1 bdev(0x2ab4c00 td/rados-striper/2/block) open stat got: (1) Operation not permitted 2026-02-21T11:39:20.374 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:20.373+0000 7f0a9fe44780 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2026-02-21T11:39:20.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/2/keyring 2026-02-21T11:39:20.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:39:20.703 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-21T11:39:20.703 INFO:tasks.workunit.client.0.trial189.stdout:adding osd2 key to auth repository 2026-02-21T11:39:20.703 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:39:20.801 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-21T11:39:20.802 INFO:tasks.workunit.client.0.trial189.stdout:start osd.2 2026-02-21T11:39:20.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=f606df79-e3dc-43f9-ab2a-a1d2bb965573 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.113465/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:39:20.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:39:20.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:39:20.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:39:20.814 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:20.813+0000 7fa671c2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:20.815 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:20.815+0000 7fa671c2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:20.816 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:20.816+0000 7fa671c2c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:20.894 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-21T11:39:20.894 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:39:20.894 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-21T11:39:20.894 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:39:20.894 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:39:20.894 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:20.894 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:39:20.894 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:20.895 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:20.895 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T11:39:20.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:39:21.003 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:21.003+0000 7fa671c2c780 -1 Falling back to public interface 2026-02-21T11:39:21.248 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:21.247+0000 7fa671c2c780 -1 osd.2 0 log_to_monitors true 2026-02-21T11:39:21.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:39:21.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:21.989 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:39:21.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:39:21.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:21.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-21T11:39:22.083 INFO:tasks.workunit.client.0.trial189.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/2818719706,v1:127.0.0.1:6817/2818719706] [v2:127.0.0.1:6818/2818719706,v1:127.0.0.1:6819/2818719706] exists,up 93b1e47d-0baf-437b-b434-beed4bc83c9c 2026-02-21T11:39:22.084 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:39:22.084 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:39:22.084 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:39:22.084 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2026-02-21T11:39:22.084 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:39:22.174 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:39:22.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:39:22.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:39:22.330 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:39:22.336 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:39:23.337 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:39:23.616 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2026-02-21T11:39:23.617 INFO:tasks.workunit.client.0.trial189.stderr:1+0 records in 2026-02-21T11:39:23.617 INFO:tasks.workunit.client.0.trial189.stderr:1+0 records out 2026-02-21T11:39:23.617 INFO:tasks.workunit.client.0.trial189.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 4.9602e-05 s, 24.9 MB/s 2026-02-21T11:39:23.617 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2026-02-21T11:39:23.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2026-02-21T11:39:23.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2026-02-21T11:39:23.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2026-02-21T11:39:23.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2026-02-21T11:39:23.666 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2026-02-21T11:39:23.666 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2026-02-21T11:39:23.666 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2026-02-21T11:39:23.667 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2026-02-21T11:39:23.682 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2026-02-21T11:39:23.683 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2026-02-21T11:39:23.699 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2026-02-21T11:39:23.699 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2026-02-21T11:39:23.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2026-02-21T11:39:23.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2026-02-21T11:39:23.733 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2026-02-21T11:39:23.733 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2026-02-21T11:39:23.747 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2026-02-21T11:39:23.747 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2026-02-21T11:39:23.761 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2026-02-21T11:39:23.761 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2026-02-21T11:39:23.762 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2026-02-21T11:39:23.762 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2026-02-21T11:39:23.777 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2026-02-21T11:39:23.792 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2026-02-21T11:39:23.807 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2026-02-21T11:39:23.807 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2026-02-21T11:39:23.808 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2026-02-21T11:39:23.808 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2026-02-21T11:39:23.822 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2026-02-21T11:39:23.823 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2026-02-21T11:39:23.823 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2026-02-21T11:39:23.823 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2026-02-21T11:39:23.837 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2026-02-21T11:39:23.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2026-02-21T11:39:23.853 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2026-02-21T11:39:23.853 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2026-02-21T11:39:23.854 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2026-02-21T11:39:23.854 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2026-02-21T11:39:23.855 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2045: expect_failure: local dir=td/rados-striper 2026-02-21T11:39:23.855 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2046: expect_failure: shift 2026-02-21T11:39:23.855 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2047: expect_failure: local 'expected=No data available' 2026-02-21T11:39:23.855 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: expect_failure: shift 2026-02-21T11:39:23.855 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2049: expect_failure: local success 2026-02-21T11:39:23.855 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2026-02-21T11:39:23.868 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: expect_failure: success=false 2026-02-21T11:39:23.869 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: expect_failure: false 2026-02-21T11:39:23.869 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2026-02-21T11:39:23.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: expect_failure: return 0 2026-02-21T11:39:23.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2026-02-21T11:39:23.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2045: expect_failure: local dir=td/rados-striper 2026-02-21T11:39:23.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2046: expect_failure: shift 2026-02-21T11:39:23.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2047: expect_failure: local 'expected=No data available' 2026-02-21T11:39:23.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: expect_failure: shift 2026-02-21T11:39:23.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2049: expect_failure: local success 2026-02-21T11:39:23.871 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2026-02-21T11:39:23.883 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: expect_failure: success=false 2026-02-21T11:39:23.883 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: expect_failure: false 2026-02-21T11:39:23.883 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2026-02-21T11:39:23.884 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: expect_failure: return 0 2026-02-21T11:39:23.884 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2026-02-21T11:39:23.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2026-02-21T11:39:23.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2045: expect_failure: local dir=td/rados-striper 2026-02-21T11:39:23.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2046: expect_failure: shift 2026-02-21T11:39:23.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2047: expect_failure: local 'expected=No such file or directory' 2026-02-21T11:39:23.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: expect_failure: shift 2026-02-21T11:39:23.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2049: expect_failure: local success 2026-02-21T11:39:23.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: expect_failure: rados --pool rbd --striper stat toyfile 2026-02-21T11:39:23.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: expect_failure: success=false 2026-02-21T11:39:23.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: expect_failure: false 2026-02-21T11:39:23.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2026-02-21T11:39:23.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: expect_failure: return 0 2026-02-21T11:39:23.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2026-02-21T11:39:23.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2045: expect_failure: local dir=td/rados-striper 2026-02-21T11:39:23.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2046: expect_failure: shift 2026-02-21T11:39:23.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2047: expect_failure: local 'expected=No such file or directory' 2026-02-21T11:39:23.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: expect_failure: shift 2026-02-21T11:39:23.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2049: expect_failure: local success 2026-02-21T11:39:23.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2026-02-21T11:39:23.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: expect_failure: success=false 2026-02-21T11:39:23.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: expect_failure: false 2026-02-21T11:39:23.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2026-02-21T11:39:23.929 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: expect_failure: return 0 2026-02-21T11:39:23.929 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2026-02-21T11:39:23.929 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2026-02-21T11:39:23.929 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:23.929 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2026-02-21T11:39:23.929 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:23.929 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:23.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:23.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:23.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:24.036 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:24.036 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:24.037 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:24.037 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:24.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:24.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:24.039 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:24.040 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:24.040 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:24.040 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:24.041 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:24.041 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:24.042 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:24.042 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2026-02-21T11:39:24.048 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:24.049 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:24.049 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.113465 2026-02-21T11:39:24.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.113465 2026-02-21T11:39:24.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:24.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:24.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: main: code=0 2026-02-21T11:39:24.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2433: main: teardown td/rados-striper 0 2026-02-21T11:39:24.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2026-02-21T11:39:24.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-21T11:39:24.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2026-02-21T11:39:24.051 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:24.051 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:24.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:24.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:24.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:24.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:24.052 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:24.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:24.053 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:24.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:24.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:24.054 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:24.055 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:24.055 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:24.055 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:24.055 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:24.056 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:24.057 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-21T11:39:24.057 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2026-02-21T11:39:24.058 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:24.058 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:24.059 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.113465 2026-02-21T11:39:24.059 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.113465 2026-02-21T11:39:24.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:24.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:24.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2434: main: return 0 2026-02-21T11:39:24.061 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-21T11:39:24.061 DEBUG:teuthology.orchestra.run.trial189:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-21T11:39:24.128 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2026-02-21T11:39:24.128 DEBUG:teuthology.orchestra.run.trial189:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2026-02-21T11:39:24.193 INFO:tasks.workunit.client.0.trial189.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-21T11:39:24.196 INFO:tasks.workunit.client.0.trial189.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-21T11:39:24.196 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: run_tests: export PATH=./bin:.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-21T11:39:24.196 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2443: run_tests: PATH=./bin:.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2026-02-21T11:39:24.196 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: run_tests: export CEPH_MON=127.0.0.1:7109 2026-02-21T11:39:24.196 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2445: run_tests: CEPH_MON=127.0.0.1:7109 2026-02-21T11:39:24.196 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2446: run_tests: export CEPH_ARGS 2026-02-21T11:39:24.196 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2447: run_tests: uuidgen 2026-02-21T11:39:24.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2447: run_tests: CEPH_ARGS+=' --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none ' 2026-02-21T11:39:24.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2448: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2026-02-21T11:39:24.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: run_tests: export CEPH_CONF=/dev/null 2026-02-21T11:39:24.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2449: run_tests: CEPH_CONF=/dev/null 2026-02-21T11:39:24.198 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: run_tests: set 2026-02-21T11:39:24.198 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2026-02-21T11:39:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: run_tests: local 'funcs=test_activate_osd 2026-02-21T11:39:24.200 INFO:tasks.workunit.client.0.trial189.stderr:test_activate_osd_after_mark_down 2026-02-21T11:39:24.200 INFO:tasks.workunit.client.0.trial189.stderr:test_activate_osd_skip_benchmark 2026-02-21T11:39:24.200 INFO:tasks.workunit.client.0.trial189.stderr:test_destroy_osd 2026-02-21T11:39:24.200 INFO:tasks.workunit.client.0.trial189.stderr:test_display_logs 2026-02-21T11:39:24.200 INFO:tasks.workunit.client.0.trial189.stderr:test_erasure_code_plugin_exists 2026-02-21T11:39:24.201 INFO:tasks.workunit.client.0.trial189.stderr:test_expect_failure 2026-02-21T11:39:24.201 INFO:tasks.workunit.client.0.trial189.stderr:test_file_with_random_data 2026-02-21T11:39:24.201 INFO:tasks.workunit.client.0.trial189.stderr:test_flush_pg_stats 2026-02-21T11:39:24.201 INFO:tasks.workunit.client.0.trial189.stderr:test_get_config 2026-02-21T11:39:24.201 INFO:tasks.workunit.client.0.trial189.stderr:test_get_is_making_recovery_progress 2026-02-21T11:39:24.201 INFO:tasks.workunit.client.0.trial189.stderr:test_get_last_scrub_stamp 2026-02-21T11:39:24.201 INFO:tasks.workunit.client.0.trial189.stderr:test_get_not_primary 2026-02-21T11:39:24.201 INFO:tasks.workunit.client.0.trial189.stderr:test_get_num_active_clean 2026-02-21T11:39:24.201 INFO:tasks.workunit.client.0.trial189.stderr:test_get_num_active_or_peered 2026-02-21T11:39:24.201 INFO:tasks.workunit.client.0.trial189.stderr:test_get_num_pgs 2026-02-21T11:39:24.201 INFO:tasks.workunit.client.0.trial189.stderr:test_get_op_scheduler 2026-02-21T11:39:24.201 INFO:tasks.workunit.client.0.trial189.stderr:test_get_osd_id_used_by_pgs 2026-02-21T11:39:24.202 INFO:tasks.workunit.client.0.trial189.stderr:test_get_osds 2026-02-21T11:39:24.202 INFO:tasks.workunit.client.0.trial189.stderr:test_get_pg 2026-02-21T11:39:24.202 INFO:tasks.workunit.client.0.trial189.stderr:test_get_primary 2026-02-21T11:39:24.202 INFO:tasks.workunit.client.0.trial189.stderr:test_get_timeout_delays 2026-02-21T11:39:24.202 INFO:tasks.workunit.client.0.trial189.stderr:test_is_clean 2026-02-21T11:39:24.202 INFO:tasks.workunit.client.0.trial189.stderr:test_kill_daemon 2026-02-21T11:39:24.202 INFO:tasks.workunit.client.0.trial189.stderr:test_kill_daemons 2026-02-21T11:39:24.202 INFO:tasks.workunit.client.0.trial189.stderr:test_objectstore_tool 2026-02-21T11:39:24.202 INFO:tasks.workunit.client.0.trial189.stderr:test_pg_schedule_scrub 2026-02-21T11:39:24.202 INFO:tasks.workunit.client.0.trial189.stderr:test_pg_scrub 2026-02-21T11:39:24.202 INFO:tasks.workunit.client.0.trial189.stderr:test_repair 2026-02-21T11:39:24.202 INFO:tasks.workunit.client.0.trial189.stderr:test_run_in_background 2026-02-21T11:39:24.202 INFO:tasks.workunit.client.0.trial189.stderr:test_run_mon 2026-02-21T11:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr:test_run_osd 2026-02-21T11:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr:test_set_config 2026-02-21T11:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr:test_setup 2026-02-21T11:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr:test_teardown 2026-02-21T11:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr:test_wait_background 2026-02-21T11:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr:test_wait_for_clean 2026-02-21T11:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr:test_wait_for_health_ok 2026-02-21T11:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr:test_wait_for_osd 2026-02-21T11:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr:test_wait_for_peered 2026-02-21T11:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr:test_wait_for_scrub 2026-02-21T11:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr:test_wait_osd_id_used_by_pgs' 2026-02-21T11:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: run_tests: local dir=td/ceph-helpers 2026-02-21T11:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_activate_osd td/ceph-helpers 2026-02-21T11:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: test_activate_osd: local dir=td/ceph-helpers 2026-02-21T11:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: test_activate_osd: setup td/ceph-helpers 2026-02-21T11:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:24.205 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:24.206 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:24.206 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:24.206 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:24.206 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:24.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:24.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:24.207 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:24.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:24.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:24.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:24.207 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:24.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:24.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:24.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:24.208 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:24.209 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:24.209 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:24.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:24.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:24.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:24.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:39:24.210 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:39:24.210 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:24.210 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:24.210 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:39:24.211 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:39:24.211 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-21T11:39:24.211 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-21T11:39:24.212 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:39:24.212 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:39:24.212 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:881: test_activate_osd: run_mon td/ceph-helpers a 2026-02-21T11:39:24.212 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:39:24.212 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:39:24.212 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:39:24.212 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:39:24.212 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:39:24.212 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T11:39:24.228 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:39:24.229 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:24.229 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:24.229 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:24.229 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:24.229 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:24.229 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:24.229 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T11:39:24.252 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:39:24.253 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:39:24.253 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:24.253 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:24.254 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:39:24.254 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:39:24.254 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:24.254 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:24.254 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:24.254 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:24.254 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:24.254 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:24.254 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:24.254 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:24.254 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:39:24.296 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:39:24.296 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:24.296 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:24.296 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:39:24.296 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:39:24.297 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:24.297 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:24.297 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:24.297 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:24.297 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:24.297 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:24.297 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:24.297 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:24.297 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:39:24.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: test_activate_osd: run_mgr td/ceph-helpers x 2026-02-21T11:39:24.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:39:24.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:39:24.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:39:24.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:39:24.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:39:24.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:39:24.438 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:39:24.439 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:24.439 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:24.439 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:24.439 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:24.439 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:24.439 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:24.440 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:24.440 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:24.457 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:884: test_activate_osd: run_osd td/ceph-helpers 0 2026-02-21T11:39:24.457 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:39:24.457 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:39:24.457 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:39:24.457 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:39:24.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:39:24.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:39:24.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:39:24.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:39:24.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:39:24.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:39:24.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:39:24.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:39:24.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:39:24.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:39:24.458 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:39:24.458 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:24.459 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:24.459 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:24.459 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:24.459 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:24.459 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:24.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:24.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:39:24.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:39:24.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:39:24.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:39:24.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:39:24.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:39:24.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:39:24.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:39:24.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:39:24.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:39:24.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:39:24.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:39:24.460 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:39:24.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f35b2604-d2b4-4e8f-8d06-fffbd1458641 2026-02-21T11:39:24.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f35b2604-d2b4-4e8f-8d06-fffbd1458641' 2026-02-21T11:39:24.461 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 f35b2604-d2b4-4e8f-8d06-fffbd1458641 2026-02-21T11:39:24.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:39:24.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBsmZlpReXTGxAA7ABOk73gs6Qp6rcZeNa4Bg== 2026-02-21T11:39:24.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBsmZlpReXTGxAA7ABOk73gs6Qp6rcZeNa4Bg=="}' 2026-02-21T11:39:24.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f35b2604-d2b4-4e8f-8d06-fffbd1458641 -i td/ceph-helpers/0/new.json 2026-02-21T11:39:24.569 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:24.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:39:24.578 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBsmZlpReXTGxAA7ABOk73gs6Qp6rcZeNa4Bg== --osd-uuid f35b2604-d2b4-4e8f-8d06-fffbd1458641 2026-02-21T11:39:24.593 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:24.593+0000 7f049b62c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:24.595 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:24.594+0000 7f049b62c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:24.595 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:24.595+0000 7f049b62c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:24.596 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:24.595+0000 7f049b62c780 -1 bdev(0x2b28c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:39:24.596 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:24.595+0000 7f049b62c780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:39:25.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:39:25.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:39:25.009 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:39:25.010 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:39:25.010 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:39:25.111 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:39:25.112 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:39:25.113 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:39:25.113 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:39:25.113 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:39:25.113 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:39:25.125 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:25.125+0000 7f56d2212780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:25.127 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:25.126+0000 7f56d2212780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:25.128 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:25.127+0000 7f56d2212780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:25.208 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:39:25.208 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:39:25.208 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:39:25.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:39:25.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:39:25.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:25.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:39:25.209 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:25.210 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:25.210 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:25.305 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:39:25.347 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:25.347+0000 7f56d2212780 -1 Falling back to public interface 2026-02-21T11:39:25.684 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:25.683+0000 7f56d2212780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:39:26.307 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:39:26.307 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:26.308 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:39:26.309 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:39:26.309 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:26.309 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:26.405 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1322390345,v1:127.0.0.1:6801/1322390345] [v2:127.0.0.1:6802/1322390345,v1:127.0.0.1:6803/1322390345] exists,up f35b2604-d2b4-4e8f-8d06-fffbd1458641 2026-02-21T11:39:26.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:39:26.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:39:26.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:39:26.407 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: get_asok_path osd.0 2026-02-21T11:39:26.407 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:39:26.407 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:39:26.408 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:26.408 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:26.408 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:26.408 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-osd.0.asok 2026-02-21T11:39:26.408 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: CEPH_ARGS= 2026-02-21T11:39:26.408 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.118434/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T11:39:26.453 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2026-02-21T11:39:26.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:887: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2026-02-21T11:39:26.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:887: test_activate_osd: grep --quiet osd_max_backfills 2026-02-21T11:39:26.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:889: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2026-02-21T11:39:26.455 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:26.455 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:26.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:26.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:26.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:26.768 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:26.768 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2026-02-21T11:39:26.768 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-21T11:39:26.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T11:39:26.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-21T11:39:26.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T11:39:26.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:39:26.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:39:26.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:39:26.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:39:26.770 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:39:26.770 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:39:26.770 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:39:26.770 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T11:39:26.770 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T11:39:26.770 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-max-backfills 20' 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:39:26.771 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:39:26.772 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-21T11:39:26.772 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2026-02-21T11:39:26.772 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-21T11:39:26.772 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-21T11:39:26.772 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T11:39:26.772 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T11:39:26.772 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T11:39:26.779 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:26.778+0000 7f8cb440e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:26.780 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:26.780+0000 7f8cb440e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:26.781 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:26.781+0000 7f8cb440e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:26.912 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:26.911+0000 7f8cb440e780 -1 Falling back to public interface 2026-02-21T11:39:26.994 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-21T11:39:26.994 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:39:26.994 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:39:26.994 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:39:26.994 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:39:26.994 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:26.994 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:39:26.994 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:26.994 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:26.994 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:27.224 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:39:27.233 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:27.233+0000 7f8cb440e780 -1 osd.0 5 log_to_monitors true 2026-02-21T11:39:28.225 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:39:28.226 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:28.226 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:39:28.227 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:39:28.227 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:28.227 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:28.450 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6800/4119429832,v1:127.0.0.1:6801/4119429832] [v2:127.0.0.1:6802/4119429832,v1:127.0.0.1:6803/4119429832] exists,up f35b2604-d2b4-4e8f-8d06-fffbd1458641 2026-02-21T11:39:28.451 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:39:28.452 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:39:28.452 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:39:28.452 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: get_asok_path osd.0 2026-02-21T11:39:28.452 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:39:28.452 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:39:28.452 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:28.452 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:28.452 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:28.452 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-osd.0.asok 2026-02-21T11:39:28.452 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: CEPH_ARGS= 2026-02-21T11:39:28.452 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.118434/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T11:39:28.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2026-02-21T11:39:28.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:894: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2026-02-21T11:39:28.499 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: teardown td/ceph-helpers 2026-02-21T11:39:28.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:28.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:28.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:28.500 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:28.500 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:28.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:28.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:28.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:28.605 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:28.606 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:28.607 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:28.607 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:28.608 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:28.608 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:28.608 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:28.609 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:28.609 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:28.610 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:28.610 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:28.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:28.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:28.612 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:28.615 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:28.615 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:28.615 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:28.615 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:28.617 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:28.617 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:28.617 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:39:28.617 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2026-02-21T11:39:28.617 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2026-02-21T11:39:28.617 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd_after_mark_down: setup td/ceph-helpers 2026-02-21T11:39:28.618 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:39:28.618 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:39:28.618 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:28.618 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:28.618 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:28.618 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:28.618 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:28.618 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:28.618 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:28.618 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:28.619 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:28.619 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:28.619 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:28.619 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:28.620 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:28.620 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:28.620 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:28.621 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:28.621 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:28.622 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:28.622 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:28.622 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:28.623 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:28.623 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:28.624 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:28.624 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:28.624 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:28.624 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:28.625 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:28.625 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:28.625 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:39:28.626 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:39:28.626 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:28.626 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:28.626 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:39:28.627 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:39:28.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:39:28.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:39:28.628 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:39:28.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:904: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2026-02-21T11:39:28.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:39:28.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:39:28.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:39:28.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:39:28.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:39:28.629 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T11:39:28.647 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:39:28.647 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:28.647 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:28.647 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:28.647 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:28.647 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:28.647 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:28.648 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T11:39:28.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:39:28.671 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:39:28.671 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:28.671 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:28.671 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:39:28.672 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:39:28.673 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:28.673 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:28.673 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:28.673 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:28.673 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:28.673 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:28.673 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:28.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:28.674 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:39:28.718 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:39:28.718 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:28.718 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:28.719 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:39:28.719 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:39:28.719 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:28.719 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:28.719 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:28.719 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:28.719 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:28.720 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:28.720 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:28.720 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:28.720 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:39:28.759 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2026-02-21T11:39:28.759 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:39:28.759 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:39:28.759 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:39:28.759 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:39:28.759 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:39:28.759 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:39:28.857 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:39:28.857 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:28.857 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:28.858 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:28.858 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:28.858 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:28.858 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:28.858 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:28.858 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:28.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:907: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2026-02-21T11:39:28.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:39:28.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:39:28.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:39:28.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:39:28.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:39:28.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:39:28.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:39:28.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:39:28.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:39:28.878 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:39:28.878 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:39:28.878 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:39:28.878 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:39:28.878 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:39:28.878 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:39:28.878 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:28.878 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:28.878 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:28.878 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:28.878 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:28.878 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:28.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:28.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:39:28.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:39:28.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:39:28.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:39:28.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:39:28.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:39:28.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:39:28.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:39:28.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:39:28.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:39:28.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:39:28.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:39:28.880 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:39:28.880 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=461609fe-7ba2-4754-bb14-65d44472eb47 2026-02-21T11:39:28.880 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 461609fe-7ba2-4754-bb14-65d44472eb47' 2026-02-21T11:39:28.881 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 461609fe-7ba2-4754-bb14-65d44472eb47 2026-02-21T11:39:28.881 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:39:28.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBwmZlpXPHZNBAAXPyj9cfExH5u3wrNC7sOwg== 2026-02-21T11:39:28.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBwmZlpXPHZNBAAXPyj9cfExH5u3wrNC7sOwg=="}' 2026-02-21T11:39:28.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 461609fe-7ba2-4754-bb14-65d44472eb47 -i td/ceph-helpers/0/new.json 2026-02-21T11:39:28.990 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:28.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:39:28.997 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBwmZlpXPHZNBAAXPyj9cfExH5u3wrNC7sOwg== --osd-uuid 461609fe-7ba2-4754-bb14-65d44472eb47 2026-02-21T11:39:29.012 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:29.012+0000 7f33ac62c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:29.014 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:29.013+0000 7f33ac62c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:29.015 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:29.014+0000 7f33ac62c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:29.015 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:29.014+0000 7f33ac62c780 -1 bdev(0x3812c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:39:29.015 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:29.014+0000 7f33ac62c780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:39:29.415 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:39:29.415 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:39:29.415 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:39:29.416 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:39:29.416 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:39:29.517 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:39:29.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:39:29.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:39:29.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:39:29.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:39:29.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:39:29.532 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:29.532+0000 7fd7e8c29780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:29.533 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:29.533+0000 7fd7e8c29780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:29.534 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:29.534+0000 7fd7e8c29780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:29.615 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:39:29.615 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:39:29.615 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:39:29.615 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:39:29.615 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:39:29.615 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:29.615 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:39:29.616 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:29.616 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:29.616 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:29.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:39:29.759 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:29.758+0000 7fd7e8c29780 -1 Falling back to public interface 2026-02-21T11:39:30.090 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:30.089+0000 7fd7e8c29780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:39:30.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:39:30.716 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:30.716 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:39:30.717 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:39:30.717 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:30.717 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:30.815 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2655181158,v1:127.0.0.1:6801/2655181158] [v2:127.0.0.1:6802/2655181158,v1:127.0.0.1:6803/2655181158] exists,up 461609fe-7ba2-4754-bb14-65d44472eb47 2026-02-21T11:39:30.816 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:39:30.817 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:39:30.817 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:39:30.817 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: get_asok_path osd.0 2026-02-21T11:39:30.817 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:39:30.817 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:39:30.817 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:30.817 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:30.817 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:30.817 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-osd.0.asok 2026-02-21T11:39:30.818 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: CEPH_ARGS= 2026-02-21T11:39:30.818 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.118434/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T11:39:30.864 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2026-02-21T11:39:30.864 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:910: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2026-02-21T11:39:30.864 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:910: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2026-02-21T11:39:30.864 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:912: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2026-02-21T11:39:30.865 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:30.865 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:30.865 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:30.866 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:30.866 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:31.171 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:31.171 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: ceph osd down 0 2026-02-21T11:39:31.397 INFO:tasks.workunit.client.0.trial189.stderr:osd.0 is already down. 2026-02-21T11:39:31.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: wait_for_osd down 0 2026-02-21T11:39:31.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2026-02-21T11:39:31.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:39:31.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:39:31.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:39:31.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:31.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:39:31.407 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:31.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:31.407 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2026-02-21T11:39:31.633 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6800/2655181158,v1:127.0.0.1:6801/2655181158] [v2:127.0.0.1:6802/2655181158,v1:127.0.0.1:6803/2655181158] exists 461609fe-7ba2-4754-bb14-65d44472eb47 2026-02-21T11:39:31.635 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:39:31.635 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:39:31.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:39:31.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2026-02-21T11:39:31.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-21T11:39:31.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T11:39:31.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-21T11:39:31.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T11:39:31.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:39:31.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:39:31.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:39:31.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:39:31.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:39:31.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:39:31.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:39:31.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T11:39:31.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T11:39:31.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:39:31.637 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T11:39:31.637 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:31.637 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:31.637 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:31.637 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:31.637 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:31.637 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:31.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:31.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:39:31.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:39:31.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:39:31.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:39:31.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:39:31.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:39:31.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:39:31.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T11:39:31.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-max-backfills 20' 2026-02-21T11:39:31.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:39:31.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-21T11:39:31.639 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:39:31.640 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-21T11:39:31.640 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2026-02-21T11:39:31.640 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-21T11:39:31.641 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T11:39:31.641 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T11:39:31.641 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T11:39:31.650 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:31.649+0000 7f5221217780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:31.651 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:31.650+0000 7f5221217780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:31.652 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:31.651+0000 7f5221217780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:31.778 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:31.777+0000 7f5221217780 -1 Falling back to public interface 2026-02-21T11:39:31.863 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-21T11:39:31.863 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:39:31.863 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:39:31.863 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:39:31.863 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:39:31.863 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:31.863 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:39:31.864 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:31.864 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:31.864 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:32.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:39:32.157 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:32.156+0000 7f5221217780 -1 osd.0 5 log_to_monitors true 2026-02-21T11:39:33.090 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:39:33.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:33.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:39:33.091 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:39:33.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:33.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:33.321 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6800/3584172138,v1:127.0.0.1:6801/3584172138] [v2:127.0.0.1:6802/3584172138,v1:127.0.0.1:6803/3584172138] exists,up 461609fe-7ba2-4754-bb14-65d44472eb47 2026-02-21T11:39:33.322 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:39:33.322 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:39:33.322 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:39:33.322 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2026-02-21T11:39:33.322 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:39:33.322 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:39:33.322 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:33.322 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:33.323 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:33.323 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-osd.0.asok 2026-02-21T11:39:33.323 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2026-02-21T11:39:33.323 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.118434/ceph-osd.0.asok config get osd_max_backfills 2026-02-21T11:39:33.370 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2026-02-21T11:39:33.370 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2026-02-21T11:39:33.370 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2026-02-21T11:39:33.370 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:33.370 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:33.370 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:33.370 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:33.370 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:33.371 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:33.371 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:33.371 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:33.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:33.476 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:33.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:33.477 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:33.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:33.479 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:33.479 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:33.480 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:33.480 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:33.480 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:33.481 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:33.481 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:33.482 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:33.482 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:33.486 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:33.486 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:33.486 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:33.486 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:33.488 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:33.488 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:33.488 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:39:33.488 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2026-02-21T11:39:33.488 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2026-02-21T11:39:33.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2026-02-21T11:39:33.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:39:33.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:39:33.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:33.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:33.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:33.489 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:33.489 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:33.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:33.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:33.489 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:33.490 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:33.490 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:33.490 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:33.491 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:33.491 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:33.491 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:33.492 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:33.492 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:33.492 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:33.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:33.493 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:33.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:33.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:33.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:33.496 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:33.496 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:33.496 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:33.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:33.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:33.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:33.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:39:33.498 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:39:33.498 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:33.498 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:33.498 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:39:33.499 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:39:33.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:39:33.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:39:33.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:39:33.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2026-02-21T11:39:33.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:39:33.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:39:33.501 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:39:33.501 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:39:33.501 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:39:33.501 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T11:39:33.518 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:39:33.519 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:33.519 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:33.519 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:33.519 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:33.519 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:33.520 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:33.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T11:39:33.540 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:39:33.541 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:39:33.541 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:33.541 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:33.541 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:39:33.541 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:39:33.541 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:33.541 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:33.541 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:33.542 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:33.542 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:33.542 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:33.542 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:33.542 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:33.542 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:39:33.584 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:39:33.584 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:33.584 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:33.584 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:39:33.584 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:39:33.584 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:33.585 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:33.585 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:33.585 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:33.585 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:33.585 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:33.585 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:33.585 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:33.585 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:39:33.627 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2026-02-21T11:39:33.627 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:39:33.627 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:39:33.627 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:39:33.627 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:39:33.627 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:39:33.627 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:39:33.725 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:39:33.725 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:33.725 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:33.725 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:33.725 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:33.725 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:33.725 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:33.726 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:33.727 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:33.743 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:33.747 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:33.747 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:33.747 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:33.747 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:33.748 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:33.748 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:39:33.748 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:39:33.748 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:39:33.748 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:39:33.748 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:39:33.749 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:39:33.749 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:39:33.749 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:39:33.749 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:39:33.749 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:39:33.749 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2026-02-21T11:39:33.749 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:39:33.750 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:39:33.750 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ba8ec637-66b2-4e48-9b34-0f644a78b544 2026-02-21T11:39:33.750 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ba8ec637-66b2-4e48-9b34-0f644a78b544' 2026-02-21T11:39:33.750 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 ba8ec637-66b2-4e48-9b34-0f644a78b544 2026-02-21T11:39:33.751 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:39:33.755 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB1mZlpCUb2LBAAr8SbBhEQiO4G8N8MPD/YqQ== 2026-02-21T11:39:33.755 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB1mZlpCUb2LBAAr8SbBhEQiO4G8N8MPD/YqQ=="}' 2026-02-21T11:39:33.756 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ba8ec637-66b2-4e48-9b34-0f644a78b544 -i td/ceph-helpers/0/new.json 2026-02-21T11:39:33.859 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:33.865 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:39:33.866 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQB1mZlpCUb2LBAAr8SbBhEQiO4G8N8MPD/YqQ== --osd-uuid ba8ec637-66b2-4e48-9b34-0f644a78b544 2026-02-21T11:39:33.882 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:33.881+0000 7f5dc2c14780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:33.883 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:33.883+0000 7f5dc2c14780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:33.884 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:33.884+0000 7f5dc2c14780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:33.885 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:33.884+0000 7f5dc2c14780 -1 bdev(0x4104c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:39:33.885 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:33.884+0000 7f5dc2c14780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:39:34.289 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:39:34.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:39:34.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:39:34.291 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:39:34.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:39:34.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:39:34.395 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:39:34.395 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-21T11:39:34.396 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:39:34.396 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:39:34.396 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:39:34.409 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:34.408+0000 7f3e5b456780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:34.410 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:34.409+0000 7f3e5b456780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:34.411 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:34.410+0000 7f3e5b456780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:34.491 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:39:34.492 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:39:34.492 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:39:34.492 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:39:34.492 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:39:34.492 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:34.492 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:39:34.492 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:34.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:34.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:34.589 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:39:34.615 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:34.615+0000 7f3e5b456780 -1 Falling back to public interface 2026-02-21T11:39:34.917 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:34.916+0000 7f3e5b456780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:39:35.590 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:39:35.590 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:35.591 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:39:35.591 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:39:35.591 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:35.591 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:35.688 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/674942344,v1:127.0.0.1:6801/674942344] [v2:127.0.0.1:6802/674942344,v1:127.0.0.1:6803/674942344] exists,up ba8ec637-66b2-4e48-9b34-0f644a78b544 2026-02-21T11:39:35.690 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:39:35.690 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:39:35.690 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:39:35.690 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-21T11:39:35.690 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:39:35.690 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:39:35.690 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:35.690 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:35.690 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:35.690 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-osd.0.asok 2026-02-21T11:39:35.691 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-21T11:39:35.691 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.118434/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2026-02-21T11:39:35.736 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2026-02-21T11:39:35.739 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-21T11:39:35.739 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:39:35.739 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:39:35.739 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:35.740 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:35.740 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:35.740 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-osd.0.asok 2026-02-21T11:39:35.740 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-21T11:39:35.740 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.118434/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2026-02-21T11:39:35.787 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2026-02-21T11:39:35.788 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:940: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2026-02-21T11:39:35.788 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:35.788 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:35.788 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:35.788 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:35.788 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:36.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:36.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:941: test_activate_osd_skip_benchmark: ceph osd down 0 2026-02-21T11:39:36.321 INFO:tasks.workunit.client.0.trial189.stderr:osd.0 is already down. 2026-02-21T11:39:36.329 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: wait_for_osd down 0 2026-02-21T11:39:36.329 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2026-02-21T11:39:36.329 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:39:36.329 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:39:36.329 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:39:36.329 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:36.329 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:36.330 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:39:36.330 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:36.330 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2026-02-21T11:39:36.553 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6800/674942344,v1:127.0.0.1:6801/674942344] [v2:127.0.0.1:6802/674942344,v1:127.0.0.1:6803/674942344] exists ba8ec637-66b2-4e48-9b34-0f644a78b544 2026-02-21T11:39:36.555 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:39:36.555 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:39:36.555 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:39:36.555 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:946: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-21T11:39:36.555 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-21T11:39:36.555 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T11:39:36.555 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-21T11:39:36.555 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T11:39:36.555 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:39:36.555 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-21T11:39:36.556 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:39:36.557 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-21T11:39:36.557 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-21T11:39:36.557 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-21T11:39:36.557 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T11:39:36.557 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T11:39:36.557 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T11:39:36.571 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:36.569+0000 7f81831d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:36.572 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:36.570+0000 7f81831d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:36.573 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:36.571+0000 7f81831d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:36.707 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:36.705+0000 7f81831d8780 -1 Falling back to public interface 2026-02-21T11:39:36.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-21T11:39:36.784 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:39:36.784 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:39:36.784 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:39:36.784 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:39:36.784 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:36.784 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:36.785 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:39:36.785 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:36.785 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:37.012 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:39:37.078 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:37.076+0000 7f81831d8780 -1 osd.0 6 log_to_monitors true 2026-02-21T11:39:38.013 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:39:38.013 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:38.013 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:39:38.013 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:39:38.013 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:38.013 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:38.237 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 6 last_clean_interval [5,6) [v2:127.0.0.1:6800/3794384121,v1:127.0.0.1:6801/3794384121] [v2:127.0.0.1:6802/3794384121,v1:127.0.0.1:6803/3794384121] exists,up ba8ec637-66b2-4e48-9b34-0f644a78b544 2026-02-21T11:39:38.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:39:38.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:39:38.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:39:38.238 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-21T11:39:38.238 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:39:38.238 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:39:38.239 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:38.239 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:38.239 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:38.239 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-osd.0.asok 2026-02-21T11:39:38.239 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-21T11:39:38.239 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.118434/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2026-02-21T11:39:38.285 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2026-02-21T11:39:38.286 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-21T11:39:38.286 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:39:38.286 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:39:38.286 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:38.286 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:38.286 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:38.286 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-osd.0.asok 2026-02-21T11:39:38.287 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-21T11:39:38.287 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.118434/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2026-02-21T11:39:38.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2026-02-21T11:39:38.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:953: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2026-02-21T11:39:38.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:954: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2026-02-21T11:39:38.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2026-02-21T11:39:38.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:38.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:38.336 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:38.336 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:38.336 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:38.336 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:38.336 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:38.336 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:38.441 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:38.442 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:38.442 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:38.443 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:38.444 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:38.444 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:38.444 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:38.445 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:38.445 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:38.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:38.446 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:38.447 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:38.448 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:38.448 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:38.451 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:38.451 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:38.451 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:38.451 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:38.453 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:38.453 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:38.453 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:39:38.453 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_destroy_osd td/ceph-helpers 2026-02-21T11:39:38.453 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: test_destroy_osd: local dir=td/ceph-helpers 2026-02-21T11:39:38.453 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: test_destroy_osd: setup td/ceph-helpers 2026-02-21T11:39:38.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:39:38.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:39:38.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:38.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:38.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:38.454 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:38.454 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:38.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:38.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:38.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:38.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:38.455 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:38.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:38.456 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:38.456 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:38.456 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:38.457 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:38.457 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:38.457 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:38.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:38.458 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:38.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:38.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:38.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:38.460 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:38.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:38.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:38.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:38.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:38.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:38.462 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:39:38.463 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:39:38.463 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:38.463 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:38.463 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:39:38.464 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:39:38.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:39:38.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:39:38.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:39:38.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: test_destroy_osd: run_mon td/ceph-helpers a 2026-02-21T11:39:38.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:39:38.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:39:38.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:39:38.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:39:38.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:39:38.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T11:39:38.483 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:39:38.483 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:38.483 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:38.484 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:38.484 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:38.484 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:38.484 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:38.484 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T11:39:38.506 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:39:38.507 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:39:38.507 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:38.507 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:38.507 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:39:38.507 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:39:38.507 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:38.507 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:38.507 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:38.508 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:38.508 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:38.508 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:38.508 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:38.508 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:38.508 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:39:38.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:39:38.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:38.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:38.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:39:38.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:39:38.551 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:38.551 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:38.551 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:38.551 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:38.551 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:38.552 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:38.552 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:38.552 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:38.552 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:39:38.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: test_destroy_osd: run_mgr td/ceph-helpers x 2026-02-21T11:39:38.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:39:38.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:39:38.593 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:39:38.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:39:38.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:39:38.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:39:38.690 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:39:38.690 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:38.690 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:38.690 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:38.690 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:38.690 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:38.690 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:38.691 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:38.691 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:38.708 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: test_destroy_osd: run_osd td/ceph-helpers 0 2026-02-21T11:39:38.708 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:39:38.708 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:39:38.708 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:39:38.708 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:39:38.708 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:39:38.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:39:38.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:39:38.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:39:38.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:39:38.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:39:38.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:39:38.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:39:38.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:39:38.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:39:38.709 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:39:38.709 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:38.709 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:38.710 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:38.710 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:38.710 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:38.710 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:38.710 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:38.710 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:39:38.710 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:39:38.710 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:39:38.710 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:39:38.710 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:39:38.711 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:39:38.711 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:39:38.711 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:39:38.711 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:39:38.711 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:39:38.711 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:39:38.711 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:39:38.711 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:39:38.712 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b2a9fa3f-251e-41af-920d-c880f6330d2f 2026-02-21T11:39:38.712 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 b2a9fa3f-251e-41af-920d-c880f6330d2f 2026-02-21T11:39:38.712 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b2a9fa3f-251e-41af-920d-c880f6330d2f' 2026-02-21T11:39:38.712 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:39:38.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB6mZlpvp3GKhAAX2+V829BljqYYsxkGIBoSQ== 2026-02-21T11:39:38.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB6mZlpvp3GKhAAX2+V829BljqYYsxkGIBoSQ=="}' 2026-02-21T11:39:38.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b2a9fa3f-251e-41af-920d-c880f6330d2f -i td/ceph-helpers/0/new.json 2026-02-21T11:39:38.820 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:38.826 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:39:38.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6mZlpvp3GKhAAX2+V829BljqYYsxkGIBoSQ== --osd-uuid b2a9fa3f-251e-41af-920d-c880f6330d2f 2026-02-21T11:39:38.842 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:38.840+0000 7fd577033780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:38.844 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:38.842+0000 7fd577033780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:38.844 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:38.843+0000 7fd577033780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:38.845 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:38.843+0000 7fd577033780 -1 bdev(0x401ac00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:39:38.845 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:38.843+0000 7fd577033780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:39:39.282 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:39:39.282 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:39:39.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:39:39.283 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:39:39.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:39:39.387 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:39:39.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:39:39.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:39:39.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:39:39.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:39:39.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:39:39.401 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:39.400+0000 7f0905668780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:39.403 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:39.401+0000 7f0905668780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:39.404 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:39.402+0000 7f0905668780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:39.486 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:39:39.486 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:39:39.486 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:39:39.486 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:39:39.486 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:39:39.486 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:39.486 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:39:39.487 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:39.487 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:39.487 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:39.583 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:39:39.602 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:39.600+0000 7f0905668780 -1 Falling back to public interface 2026-02-21T11:39:39.963 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:39.961+0000 7f0905668780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:39:40.584 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:39:40.585 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:40.585 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:39:40.586 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:39:40.586 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:40.586 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:40.683 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3317697299,v1:127.0.0.1:6801/3317697299] [v2:127.0.0.1:6802/3317697299,v1:127.0.0.1:6803/3317697299] exists,up b2a9fa3f-251e-41af-920d-c880f6330d2f 2026-02-21T11:39:40.683 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:39:40.683 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:39:40.683 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:39:40.683 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:793: test_destroy_osd: destroy_osd td/ceph-helpers 0 2026-02-21T11:39:40.683 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:775: destroy_osd: local dir=td/ceph-helpers 2026-02-21T11:39:40.683 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:776: destroy_osd: local id=0 2026-02-21T11:39:40.683 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:778: destroy_osd: ceph osd out osd.0 2026-02-21T11:39:40.886 INFO:tasks.workunit.client.0.trial189.stderr:osd.0 is already out. 2026-02-21T11:39:40.892 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:779: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2026-02-21T11:39:40.893 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:40.893 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:40.893 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:40.893 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:40.893 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:41.199 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:41.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:780: destroy_osd: ceph osd down osd.0 2026-02-21T11:39:41.429 INFO:tasks.workunit.client.0.trial189.stderr:osd.0 is already down. 2026-02-21T11:39:41.436 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:781: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2026-02-21T11:39:41.655 INFO:tasks.workunit.client.0.trial189.stderr:osd.0 does not exist 2026-02-21T11:39:41.663 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:782: destroy_osd: teardown td/ceph-helpers/0 2026-02-21T11:39:41.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers/0 2026-02-21T11:39:41.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:41.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers/0 KILL 2026-02-21T11:39:41.664 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:41.664 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:41.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:41.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:41.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:41.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:41.666 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:41.666 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:41.667 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:41.668 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:41.668 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:41.668 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:41.668 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:41.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:41.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:41.669 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:41.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:41.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:41.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers/0 2026-02-21T11:39:41.672 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:41.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:41.673 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:41.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:41.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:41.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:41.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:783: destroy_osd: rm -fr td/ceph-helpers/0 2026-02-21T11:39:41.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:794: test_destroy_osd: ceph osd dump 2026-02-21T11:39:41.675 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:794: test_destroy_osd: grep 'osd. ' 2026-02-21T11:39:41.897 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:795: test_destroy_osd: teardown td/ceph-helpers 2026-02-21T11:39:41.897 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:41.897 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:41.897 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:41.898 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:41.898 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:41.898 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:41.898 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:41.898 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:42.004 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:42.004 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:42.005 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:42.005 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:42.007 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:42.007 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:42.007 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:42.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:42.008 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:42.009 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:42.009 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:42.009 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:42.010 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:42.010 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:42.013 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:42.013 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:42.013 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:42.013 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:42.015 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:42.015 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:42.015 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:39:42.016 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_display_logs td/ceph-helpers 2026-02-21T11:39:42.016 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: test_display_logs: local dir=td/ceph-helpers 2026-02-21T11:39:42.016 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2189: test_display_logs: setup td/ceph-helpers 2026-02-21T11:39:42.016 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:39:42.016 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:39:42.016 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:42.016 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:42.016 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:42.016 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:42.016 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:42.016 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:42.017 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:42.017 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:42.017 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:42.018 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:42.018 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:42.018 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:42.018 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:42.019 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:42.019 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:42.019 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:42.020 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:42.020 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:42.020 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:42.021 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:42.021 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:42.022 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:42.023 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:42.023 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:42.023 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:42.023 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:42.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:42.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:42.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:39:42.025 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:39:42.025 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:42.025 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:42.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:39:42.026 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:39:42.027 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:39:42.027 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:39:42.027 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:39:42.027 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: test_display_logs: run_mon td/ceph-helpers a 2026-02-21T11:39:42.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:39:42.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:39:42.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:39:42.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:39:42.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:39:42.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T11:39:42.045 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:39:42.045 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:42.045 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:42.045 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:42.046 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:42.046 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:42.046 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:42.046 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T11:39:42.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:39:42.072 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:39:42.072 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:42.072 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:42.073 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:39:42.073 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:39:42.073 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:42.073 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:42.073 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:42.074 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:42.074 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:42.074 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:42.074 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:42.074 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:42.074 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:39:42.118 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:39:42.118 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:42.118 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:42.118 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:39:42.118 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:39:42.118 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:42.118 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:42.118 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:42.118 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:42.119 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:42.119 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:42.119 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:42.119 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:42.119 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:39:42.160 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2191: test_display_logs: kill_daemons td/ceph-helpers 2026-02-21T11:39:42.160 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:42.160 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:42.161 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:42.161 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:42.161 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:42.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:42.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: test_display_logs: display_logs td/ceph-helpers 2026-02-21T11:39:42.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2177: display_logs: local dir=td/ceph-helpers 2026-02-21T11:39:42.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2179: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2026-02-21T11:39:42.267 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2180: display_logs: read file 2026-02-21T11:39:42.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2026-02-21T11:39:42.268 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: display_logs: cat td/ceph-helpers/mon.a.log 2026-02-21T11:39:42.269 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2180: display_logs: read file 2026-02-21T11:39:42.269 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2193: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2026-02-21T11:39:42.271 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: test_display_logs: teardown td/ceph-helpers 2026-02-21T11:39:42.271 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:42.271 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:42.271 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:42.272 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:42.272 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:42.272 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:42.272 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:42.272 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:42.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:42.274 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:42.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:42.274 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:42.275 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:42.275 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:42.276 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:42.276 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:42.276 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:42.277 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:42.277 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:42.278 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:42.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:42.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:42.280 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:42.280 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:42.280 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:42.280 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:42.282 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:42.282 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:42.282 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:39:42.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2026-02-21T11:39:42.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2026-02-21T11:39:42.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: test_erasure_code_plugin_exists: setup td/ceph-helpers 2026-02-21T11:39:42.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:39:42.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:39:42.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:42.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:42.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:42.283 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:42.283 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:42.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:42.283 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:42.284 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:42.284 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:42.284 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:42.284 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:42.285 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:42.285 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:42.285 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:42.286 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:42.286 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:42.286 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:42.287 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:42.287 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:42.287 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:42.289 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:42.289 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:42.290 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:42.290 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:42.290 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:42.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:42.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:42.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:42.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:39:42.292 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:39:42.292 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:42.292 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:42.292 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:39:42.293 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:39:42.294 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:39:42.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:39:42.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:39:42.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2161: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2026-02-21T11:39:42.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:39:42.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:39:42.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:39:42.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:39:42.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:39:42.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T11:39:42.312 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:39:42.312 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:42.312 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:42.312 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:42.312 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:42.312 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:42.312 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:42.312 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T11:39:42.334 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:39:42.336 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:39:42.337 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:42.337 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:42.337 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:39:42.337 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:39:42.337 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:42.337 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:42.337 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:42.337 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:42.337 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:42.338 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:42.338 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:42.338 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:42.338 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:39:42.379 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:39:42.379 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:42.379 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:42.379 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:39:42.379 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:39:42.379 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:42.379 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:42.379 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:42.379 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:42.380 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:42.380 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:42.380 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:42.380 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:42.380 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:39:42.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2026-02-21T11:39:42.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:39:42.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:39:42.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:39:42.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:39:42.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:39:42.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:39:42.522 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:39:42.522 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:42.522 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:42.523 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:42.523 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:42.523 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:42.523 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:42.523 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:42.524 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:42.539 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2026-02-21T11:39:42.539 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: erasure_code_plugin_exists: local plugin=jerasure 2026-02-21T11:39:42.539 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: erasure_code_plugin_exists: local status 2026-02-21T11:39:42.539 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: erasure_code_plugin_exists: local grepstr 2026-02-21T11:39:42.539 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: erasure_code_plugin_exists: local s 2026-02-21T11:39:42.539 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: erasure_code_plugin_exists: case `uname` in 2026-02-21T11:39:42.539 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: erasure_code_plugin_exists: uname 2026-02-21T11:39:42.540 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2026-02-21T11:39:42.540 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2026-02-21T11:39:42.646 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: erasure_code_plugin_exists: s= 2026-02-21T11:39:42.646 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: erasure_code_plugin_exists: local status=0 2026-02-21T11:39:42.647 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2026-02-21T11:39:42.647 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2026-02-21T11:39:42.822 INFO:tasks.workunit.client.0.trial189.stderr:erasure-code-profile TESTPROFILE does not exist 2026-02-21T11:39:42.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: erasure_code_plugin_exists: return 0 2026-02-21T11:39:42.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2164: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2026-02-21T11:39:42.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: erasure_code_plugin_exists: local plugin=FAKE 2026-02-21T11:39:42.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: erasure_code_plugin_exists: local status 2026-02-21T11:39:42.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: erasure_code_plugin_exists: local grepstr 2026-02-21T11:39:42.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: erasure_code_plugin_exists: local s 2026-02-21T11:39:42.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: erasure_code_plugin_exists: case `uname` in 2026-02-21T11:39:42.828 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: erasure_code_plugin_exists: uname 2026-02-21T11:39:42.829 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2026-02-21T11:39:42.830 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2026-02-21T11:39:42.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so): /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2026-02-21T11:39:42.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: erasure_code_plugin_exists: local status=5 2026-02-21T11:39:42.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2026-02-21T11:39:42.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib64/ceph/erasure-code/libec_FAKE.so):' /usr/lib64/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2026-02-21T11:39:42.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2026-02-21T11:39:42.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: erasure_code_plugin_exists: status=1 2026-02-21T11:39:42.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: erasure_code_plugin_exists: return 1 2026-02-21T11:39:42.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2026-02-21T11:39:42.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:42.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:42.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:42.916 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:42.916 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:42.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:42.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:42.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:43.021 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:43.021 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:43.022 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:43.022 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:43.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:43.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:43.024 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:43.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:43.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:43.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:43.026 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:43.026 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:43.027 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:43.027 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:43.029 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:43.029 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:43.029 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:43.029 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:43.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:43.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:43.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:39:43.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_expect_failure td/ceph-helpers 2026-02-21T11:39:43.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: test_expect_failure: local dir=td/ceph-helpers 2026-02-21T11:39:43.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: test_expect_failure: setup td/ceph-helpers 2026-02-21T11:39:43.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:39:43.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:39:43.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:43.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:43.032 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:43.032 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:43.032 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:43.032 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:43.032 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:43.032 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:43.032 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:43.032 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:43.033 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:43.033 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:43.034 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:43.034 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:43.035 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:43.035 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:43.036 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:43.036 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:43.036 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:43.037 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:43.037 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:43.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:43.039 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:43.039 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:43.039 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:43.039 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:43.040 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:43.040 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:43.040 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:39:43.041 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:39:43.041 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:43.041 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:43.041 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:39:43.042 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:39:43.043 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:39:43.043 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:39:43.043 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:39:43.043 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2026-02-21T11:39:43.043 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2045: expect_failure: local dir=td/ceph-helpers 2026-02-21T11:39:43.043 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2046: expect_failure: shift 2026-02-21T11:39:43.043 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2047: expect_failure: local expected=FAIL 2026-02-21T11:39:43.043 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: expect_failure: shift 2026-02-21T11:39:43.043 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2049: expect_failure: local success 2026-02-21T11:39:43.043 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: expect_failure: bash -c 'echo FAIL ; exit 1' 2026-02-21T11:39:43.044 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: expect_failure: success=false 2026-02-21T11:39:43.044 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: expect_failure: false 2026-02-21T11:39:43.044 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-21T11:39:43.045 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: expect_failure: return 0 2026-02-21T11:39:43.045 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2026-02-21T11:39:43.045 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2045: expect_failure: local dir=td/ceph-helpers 2026-02-21T11:39:43.045 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2046: expect_failure: shift 2026-02-21T11:39:43.045 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2047: expect_failure: local expected=FAIL 2026-02-21T11:39:43.045 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: expect_failure: shift 2026-02-21T11:39:43.045 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2049: expect_failure: local success 2026-02-21T11:39:43.045 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: expect_failure: bash -c 'echo FAIL ; exit 0' 2026-02-21T11:39:43.046 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: expect_failure: success=true 2026-02-21T11:39:43.046 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: expect_failure: true 2026-02-21T11:39:43.046 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2058: expect_failure: cat td/ceph-helpers/out 2026-02-21T11:39:43.047 INFO:tasks.workunit.client.0.trial189.stderr:FAIL 2026-02-21T11:39:43.047 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: expect_failure: return 1 2026-02-21T11:39:43.047 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-21T11:39:43.048 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2026-02-21T11:39:43.048 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2045: expect_failure: local dir=td/ceph-helpers 2026-02-21T11:39:43.048 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2046: expect_failure: shift 2026-02-21T11:39:43.048 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2047: expect_failure: local expected=FAIL 2026-02-21T11:39:43.048 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2048: expect_failure: shift 2026-02-21T11:39:43.048 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2049: expect_failure: local success 2026-02-21T11:39:43.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2026-02-21T11:39:43.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2054: expect_failure: success=false 2026-02-21T11:39:43.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: expect_failure: false 2026-02-21T11:39:43.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-21T11:39:43.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2058: expect_failure: cat td/ceph-helpers/out 2026-02-21T11:39:43.051 INFO:tasks.workunit.client.0.trial189.stderr:UNEXPECTED 2026-02-21T11:39:43.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: expect_failure: return 1 2026-02-21T11:39:43.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-21T11:39:43.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: test_expect_failure: teardown td/ceph-helpers 2026-02-21T11:39:43.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:43.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:43.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:43.052 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:43.053 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:43.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:43.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:43.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:43.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:43.054 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:43.055 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:43.055 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:43.056 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:43.056 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:43.056 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:43.057 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:43.057 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:43.058 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:43.058 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:43.058 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:43.059 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:43.059 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:43.060 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:43.060 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:43.060 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:43.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:43.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:43.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:43.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:39:43.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_file_with_random_data td/ceph-helpers 2026-02-21T11:39:43.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: test_file_with_random_data: local dir=td/ceph-helpers 2026-02-21T11:39:43.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: test_file_with_random_data: setup td/ceph-helpers 2026-02-21T11:39:43.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:39:43.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:39:43.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:43.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:43.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:43.063 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:43.064 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:43.064 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:43.064 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:43.064 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:43.065 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:43.065 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:43.065 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:43.065 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:43.066 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:43.066 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:43.066 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:43.067 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:43.067 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:43.067 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:43.067 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:43.068 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:43.068 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:43.068 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:43.069 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:43.070 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:43.070 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:43.070 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:43.070 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:43.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:43.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:39:43.072 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:39:43.072 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:43.072 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:43.072 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:39:43.073 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:39:43.074 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:39:43.074 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:39:43.074 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:39:43.074 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2382: test_file_with_random_data: file_with_random_data 4000 2026-02-21T11:39:43.074 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: file_with_random_data: local size_bytes=4000 2026-02-21T11:39:43.074 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: file_with_random_data: mktemp 2026-02-21T11:39:43.075 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: file_with_random_data: local file=/tmp/tmp.xzuYlBmsn5 2026-02-21T11:39:43.075 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: file_with_random_data: dd if=/dev/urandom of=/tmp/tmp.xzuYlBmsn5 bs=4000 count=1 2026-02-21T11:39:43.076 INFO:tasks.workunit.client.0.trial189.stderr:1+0 records in 2026-02-21T11:39:43.076 INFO:tasks.workunit.client.0.trial189.stderr:1+0 records out 2026-02-21T11:39:43.076 INFO:tasks.workunit.client.0.trial189.stderr:4000 bytes (4.0 kB, 3.9 KiB) copied, 0.00010692 s, 37.4 MB/s 2026-02-21T11:39:43.076 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: file_with_random_data: printf %s /tmp/tmp.xzuYlBmsn5 2026-02-21T11:39:43.076 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2382: test_file_with_random_data: local file=/tmp/tmp.xzuYlBmsn5 2026-02-21T11:39:43.076 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: test_file_with_random_data: test -f /tmp/tmp.xzuYlBmsn5 2026-02-21T11:39:43.076 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: test_file_with_random_data: stat -c %s /tmp/tmp.xzuYlBmsn5 2026-02-21T11:39:43.077 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: test_file_with_random_data: test 4000 = 4000 2026-02-21T11:39:43.077 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: test_file_with_random_data: rm /tmp/tmp.xzuYlBmsn5 2026-02-21T11:39:43.078 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: test_file_with_random_data: teardown td/ceph-helpers 2026-02-21T11:39:43.078 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:43.078 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:43.078 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:43.079 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:43.079 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:43.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:43.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:43.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:43.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:43.080 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:43.081 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:43.081 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:43.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:43.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:43.082 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:43.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:43.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:43.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:43.083 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:43.084 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:43.085 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:43.085 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:43.086 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:43.086 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:43.086 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:43.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:43.088 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:43.088 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:43.088 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:39:43.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_flush_pg_stats td/ceph-helpers 2026-02-21T11:39:43.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: test_flush_pg_stats: local dir=td/ceph-helpers 2026-02-21T11:39:43.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: test_flush_pg_stats: setup td/ceph-helpers 2026-02-21T11:39:43.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:39:43.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:39:43.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:43.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:43.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:43.089 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:43.089 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:43.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:43.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:43.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:43.090 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:43.090 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:43.090 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:43.091 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:43.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:43.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:43.092 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:43.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:43.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:43.093 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:43.093 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:43.093 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:43.094 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:43.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:43.096 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:43.096 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:43.096 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:43.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:43.097 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:43.097 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:43.097 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:39:43.098 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:39:43.098 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:43.098 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:43.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:39:43.099 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:39:43.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:39:43.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:39:43.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:39:43.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2312: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:39:43.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:39:43.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:39:43.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:39:43.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:39:43.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:39:43.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:39:43.118 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:39:43.118 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:43.118 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:43.118 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:43.118 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:43.118 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:43.118 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:43.118 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:39:43.143 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:39:43.144 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:39:43.144 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:43.144 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:43.144 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:39:43.145 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:39:43.145 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:43.145 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:43.145 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:43.145 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:43.145 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:43.145 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:43.145 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:43.146 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:43.146 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:39:43.190 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:39:43.190 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:43.191 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:43.191 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:39:43.191 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:39:43.191 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:43.191 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:43.191 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:43.192 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:43.192 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:43.192 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:43.192 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:43.192 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:43.192 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:39:43.234 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2313: test_flush_pg_stats: run_mgr td/ceph-helpers x 2026-02-21T11:39:43.234 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:39:43.234 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:39:43.234 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:39:43.234 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:39:43.234 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:39:43.234 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:39:43.335 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:39:43.335 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:43.335 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:43.335 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:43.335 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:43.335 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:43.335 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:43.335 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:43.336 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:43.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2314: test_flush_pg_stats: run_osd td/ceph-helpers 0 2026-02-21T11:39:43.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:39:43.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:39:43.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:39:43.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:39:43.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:39:43.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:39:43.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:39:43.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:39:43.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:39:43.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:39:43.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:39:43.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:39:43.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:39:43.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:39:43.355 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:39:43.356 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:43.356 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:43.356 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:43.356 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:43.356 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:43.356 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:43.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:43.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:39:43.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:39:43.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:39:43.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:39:43.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:39:43.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:39:43.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:39:43.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:39:43.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:39:43.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:39:43.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:39:43.357 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:39:43.357 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:39:43.358 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=31c2b840-f0ea-44ed-a45f-92a1732b0c30 2026-02-21T11:39:43.358 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 31c2b840-f0ea-44ed-a45f-92a1732b0c30' 2026-02-21T11:39:43.358 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 31c2b840-f0ea-44ed-a45f-92a1732b0c30 2026-02-21T11:39:43.358 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:39:43.366 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB/mZlp/625FRAAX0BcWuwjUP/HDER54PSq9Q== 2026-02-21T11:39:43.366 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB/mZlp/625FRAAX0BcWuwjUP/HDER54PSq9Q=="}' 2026-02-21T11:39:43.366 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 31c2b840-f0ea-44ed-a45f-92a1732b0c30 -i td/ceph-helpers/0/new.json 2026-02-21T11:39:43.468 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:43.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:39:43.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB/mZlp/625FRAAX0BcWuwjUP/HDER54PSq9Q== --osd-uuid 31c2b840-f0ea-44ed-a45f-92a1732b0c30 2026-02-21T11:39:43.491 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:43.489+0000 7ffaac425780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:43.492 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:43.490+0000 7ffaac425780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:43.492 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:43.491+0000 7ffaac425780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:43.493 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:43.491+0000 7ffaac425780 -1 bdev(0x3336c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:39:43.493 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:43.491+0000 7ffaac425780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:39:43.883 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:39:43.884 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:39:43.884 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:39:43.885 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:39:43.885 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:39:43.989 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:39:43.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:39:43.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:39:43.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:39:43.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:39:43.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:39:44.004 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:44.002+0000 7ff0e8842780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:44.005 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:44.003+0000 7ff0e8842780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:44.006 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:44.004+0000 7ff0e8842780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:44.090 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:39:44.090 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:39:44.090 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:39:44.090 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:39:44.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:39:44.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:44.091 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:44.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:39:44.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:44.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:44.190 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:39:44.215 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:44.214+0000 7ff0e8842780 -1 Falling back to public interface 2026-02-21T11:39:44.553 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:44.551+0000 7ff0e8842780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:39:45.191 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:39:45.192 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:45.192 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:39:45.193 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:39:45.193 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:45.193 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:45.289 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2093801545,v1:127.0.0.1:6801/2093801545] [v2:127.0.0.1:6802/2093801545,v1:127.0.0.1:6803/2093801545] exists,up 31c2b840-f0ea-44ed-a45f-92a1732b0c30 2026-02-21T11:39:45.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:39:45.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:39:45.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:39:45.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: test_flush_pg_stats: create_rbd_pool 2026-02-21T11:39:45.291 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:39:45.383 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:39:45.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:39:45.390 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:39:45.548 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:39:45.553 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:39:46.555 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:39:46.832 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2316: test_flush_pg_stats: rados -p rbd put obj /etc/group 2026-02-21T11:39:46.848 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: test_flush_pg_stats: flush_pg_stats 2026-02-21T11:39:46.849 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:39:46.849 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:39:47.074 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:39:47.074 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:39:47.074 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:47.074 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:39:47.138 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836483 2026-02-21T11:39:47.138 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836483 2026-02-21T11:39:47.138 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T11:39:47.138 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:47.138 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836483 2026-02-21T11:39:47.138 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:47.139 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:39:47.140 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836483 2026-02-21T11:39:47.140 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:47.141 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836483 2026-02-21T11:39:47.141 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T11:39:47.141 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836483 2026-02-21T11:39:47.141 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:39:47.366 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T11:39:47.367 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:39:48.367 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:39:48.367 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:39:48.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T11:39:48.600 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2318: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2026-02-21T11:39:48.600 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: test_flush_pg_stats: ceph df detail --format=json 2026-02-21T11:39:48.601 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2026-02-21T11:39:48.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: test_flush_pg_stats: stored=8189 2026-02-21T11:39:48.916 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: test_flush_pg_stats: ceph df detail --format=json 2026-02-21T11:39:48.916 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2026-02-21T11:39:49.228 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: test_flush_pg_stats: stored_raw=8189 2026-02-21T11:39:49.228 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2321: test_flush_pg_stats: test 8189 -gt 0 2026-02-21T11:39:49.228 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: test_flush_pg_stats: test 8189 == 8189 2026-02-21T11:39:49.229 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: test_flush_pg_stats: teardown td/ceph-helpers 2026-02-21T11:39:49.229 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:49.229 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:49.229 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:49.229 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:49.229 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:49.229 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:49.229 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:49.229 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:49.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:49.336 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:49.336 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:49.337 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:49.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:49.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:49.338 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:49.339 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:49.339 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:49.340 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:49.340 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:49.341 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:49.342 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:49.342 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:49.345 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:49.345 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:49.346 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:49.346 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_get_config td/ceph-helpers 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: test_get_config: local dir=td/ceph-helpers 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: test_get_config: setup td/ceph-helpers 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:49.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:49.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:49.349 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:49.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:49.350 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:49.351 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:49.351 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:49.351 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:49.352 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:49.352 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:49.352 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:49.352 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:49.353 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:49.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:49.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:49.355 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:49.355 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:49.355 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:49.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:49.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:49.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:49.356 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:39:49.357 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:39:49.357 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:49.357 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:49.358 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:39:49.359 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:39:49.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:39:49.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:39:49.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:39:49.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1131: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:39:49.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:39:49.359 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:39:49.360 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:39:49.360 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:39:49.360 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:39:49.360 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:39:49.377 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:39:49.377 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:49.377 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:49.377 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:49.377 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:49.377 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:49.377 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:49.378 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:39:49.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:39:49.402 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:39:49.402 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:49.402 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:49.402 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:39:49.402 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:39:49.402 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:49.402 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:49.403 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:49.403 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:49.403 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:49.403 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:49.403 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:49.403 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:49.403 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:39:49.445 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:39:49.445 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:49.445 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:49.445 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:39:49.445 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:39:49.445 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:49.445 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:49.445 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:49.446 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:49.446 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:49.446 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:49.446 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:49.446 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:49.446 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:39:49.487 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: test_get_config: get_config mon a osd_pool_default_size 2026-02-21T11:39:49.488 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:49.488 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:49.488 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_pool_default_size 2026-02-21T11:39:49.488 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_pool_default_size 2026-02-21T11:39:49.488 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:49.488 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:49.488 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:49.488 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:49.488 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:49.488 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:49.488 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:49.489 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:49.489 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get osd_pool_default_size 2026-02-21T11:39:49.531 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: test_get_config: test 1 = 1 2026-02-21T11:39:49.531 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: test_get_config: run_mgr td/ceph-helpers x 2026-02-21T11:39:49.531 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:39:49.532 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:39:49.532 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:39:49.532 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:39:49.532 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:39:49.532 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:39:49.634 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:39:49.634 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:49.634 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:49.634 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:49.634 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:49.634 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:49.634 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:49.635 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:49.635 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:49.651 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2026-02-21T11:39:49.651 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:39:49.651 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:39:49.651 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:39:49.651 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:39:49.651 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:39:49.651 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:39:49.651 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:39:49.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:39:49.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:39:49.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:39:49.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:39:49.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:39:49.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:39:49.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:39:49.652 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:39:49.652 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:49.652 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:49.654 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:49.654 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:49.655 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:49.655 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:49.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:49.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:39:49.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:39:49.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:39:49.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:39:49.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:39:49.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:39:49.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:39:49.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:39:49.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:39:49.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:39:49.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_max_scrubs=3 2026-02-21T11:39:49.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:39:49.656 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:39:49.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=493df3c6-a277-4c82-8730-23a5bd06ba3f 2026-02-21T11:39:49.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 493df3c6-a277-4c82-8730-23a5bd06ba3f' 2026-02-21T11:39:49.657 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 493df3c6-a277-4c82-8730-23a5bd06ba3f 2026-02-21T11:39:49.657 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:39:49.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCFmZlpGDmGJxAAdxGriZ0VsZcGyoJm6fgqTw== 2026-02-21T11:39:49.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCFmZlpGDmGJxAAdxGriZ0VsZcGyoJm6fgqTw=="}' 2026-02-21T11:39:49.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 493df3c6-a277-4c82-8730-23a5bd06ba3f -i td/ceph-helpers/0/new.json 2026-02-21T11:39:49.766 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:49.773 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:39:49.773 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQCFmZlpGDmGJxAAdxGriZ0VsZcGyoJm6fgqTw== --osd-uuid 493df3c6-a277-4c82-8730-23a5bd06ba3f 2026-02-21T11:39:49.789 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:49.788+0000 7f350e53d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:49.791 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:49.789+0000 7f350e53d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:49.791 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:49.790+0000 7f350e53d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:49.792 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:49.790+0000 7f350e53d780 -1 bdev(0x281cc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:39:49.792 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:49.790+0000 7f350e53d780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:39:50.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:39:50.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:39:50.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:39:50.198 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:39:50.198 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:39:50.298 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:39:50.299 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:39:50.300 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2026-02-21T11:39:50.300 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:39:50.300 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:39:50.300 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:39:50.313 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:50.311+0000 7f9a3b668780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:50.314 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:50.313+0000 7f9a3b668780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:50.315 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:50.314+0000 7f9a3b668780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:50.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:39:50.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:39:50.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:39:50.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:39:50.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:39:50.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:50.397 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:50.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:39:50.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:50.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:50.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:39:50.530 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:50.528+0000 7f9a3b668780 -1 Falling back to public interface 2026-02-21T11:39:50.907 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:50.906+0000 7f9a3b668780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:39:51.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:39:51.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:51.495 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:39:51.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:39:51.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:51.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:51.595 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/871033501,v1:127.0.0.1:6801/871033501] [v2:127.0.0.1:6802/871033501,v1:127.0.0.1:6803/871033501] exists,up 493df3c6-a277-4c82-8730-23a5bd06ba3f 2026-02-21T11:39:51.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:39:51.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:39:51.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:39:51.598 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: test_get_config: get_config osd 0 osd_max_scrubs 2026-02-21T11:39:51.598 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2026-02-21T11:39:51.598 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=0 2026-02-21T11:39:51.598 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_max_scrubs 2026-02-21T11:39:51.598 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_max_scrubs 2026-02-21T11:39:51.598 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.0 2026-02-21T11:39:51.598 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:39:51.598 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:39:51.598 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:51.598 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:51.598 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:51.598 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-osd.0.asok 2026-02-21T11:39:51.599 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:51.599 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-osd.0.asok config get osd_max_scrubs 2026-02-21T11:39:51.646 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: test_get_config: test 3 = 3 2026-02-21T11:39:51.647 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: teardown td/ceph-helpers 2026-02-21T11:39:51.647 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:51.647 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:51.647 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:51.647 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:51.647 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:51.647 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:51.647 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:51.647 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:51.753 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:51.753 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:51.754 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:51.754 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:51.756 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:51.756 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:51.756 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:51.757 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:51.757 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:51.757 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:51.758 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:51.758 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:51.759 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:51.759 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:51.763 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:51.763 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:51.763 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:51.763 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:51.766 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:51.766 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:51.766 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:39:51.766 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2026-02-21T11:39:51.766 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2026-02-21T11:39:51.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: test_get_is_making_recovery_progress: setup td/ceph-helpers 2026-02-21T11:39:51.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:39:51.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:39:51.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:51.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:51.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:51.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:51.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:51.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:51.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:51.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:51.767 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:51.768 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:51.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:51.769 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:51.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:51.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:51.769 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:51.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:51.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:51.769 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:51.770 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:51.770 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:51.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:51.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:51.772 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:51.772 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:51.772 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:51.772 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:51.773 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:51.773 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:51.773 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:39:51.774 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:39:51.774 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:51.774 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:51.774 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:39:51.775 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:39:51.776 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:39:51.776 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:39:51.776 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:39:51.776 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2026-02-21T11:39:51.776 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:39:51.777 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:39:51.777 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:39:51.777 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:39:51.777 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:39:51.777 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T11:39:51.795 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:39:51.795 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:51.796 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:51.796 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:51.796 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:51.796 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:51.796 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:51.796 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T11:39:51.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:39:51.820 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:39:51.820 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:51.820 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:51.821 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:39:51.821 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:39:51.821 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:51.821 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:51.821 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:51.821 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:51.821 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:51.821 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:51.821 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:51.821 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:51.821 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:39:51.867 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:39:51.867 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:51.867 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:51.867 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:39:51.867 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:39:51.867 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:51.868 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:51.868 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:51.868 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:51.868 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:51.868 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:51.868 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:51.869 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:51.869 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:39:51.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2026-02-21T11:39:51.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:39:51.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:39:51.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:39:51.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:39:51.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:39:51.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:39:52.008 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:39:52.008 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:52.008 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:52.008 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:52.009 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:52.009 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:52.009 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:52.009 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:52.010 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:52.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2026-02-21T11:39:52.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-21T11:39:52.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-21T11:39:52.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-21T11:39:52.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-21T11:39:52.026 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-21T11:39:52.026 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-21T11:39:52.125 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-21T11:39:52.125 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-21T11:39:52.125 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1341: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2026-02-21T11:39:52.125 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:52.125 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:52.125 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:52.125 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:52.125 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:52.125 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:52.125 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:52.126 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:52.231 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:52.232 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:52.232 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:52.233 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:52.234 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:52.234 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:52.234 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:52.235 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:52.235 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:52.236 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:52.236 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:52.237 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:52.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:52.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:52.240 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:52.240 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:52.240 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:52.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:52.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:52.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:52.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:39:52.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2026-02-21T11:39:52.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1524: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2026-02-21T11:39:52.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: test_get_last_scrub_stamp: setup td/ceph-helpers 2026-02-21T11:39:52.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:39:52.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:39:52.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:39:52.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:39:52.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:39:52.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:39:52.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:39:52.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:39:52.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:39:52.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:39:52.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:39:52.244 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:39:52.244 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:39:52.244 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:39:52.245 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:39:52.245 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:39:52.245 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:39:52.246 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:52.246 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:39:52.246 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:39:52.246 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:39:52.247 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:39:52.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:39:52.248 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:39:52.249 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:39:52.249 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:52.249 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:52.249 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:39:52.250 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:39:52.250 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:39:52.250 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:39:52.251 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:39:52.251 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:52.251 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:52.251 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:39:52.252 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:39:52.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:39:52.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:39:52.253 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:39:52.254 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:39:52.254 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:39:52.254 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:39:52.254 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:39:52.254 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:39:52.254 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:39:52.254 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:39:52.272 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:39:52.272 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:52.272 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:52.272 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:52.272 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:52.272 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:52.272 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:52.272 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:39:52.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:39:52.297 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:39:52.297 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:52.298 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:52.298 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:39:52.298 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:39:52.298 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:52.298 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:52.298 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:52.298 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:52.298 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:52.298 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:52.298 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:52.298 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:52.298 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:39:52.340 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:39:52.340 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:39:52.341 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:39:52.341 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:39:52.341 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:39:52.341 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:39:52.341 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:39:52.341 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:39:52.341 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:39:52.341 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:52.341 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:52.341 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:39:52.341 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:39:52.341 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:39:52.381 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2026-02-21T11:39:52.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:39:52.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:39:52.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:39:52.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:39:52.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:39:52.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:39:52.477 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:39:52.477 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:52.477 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:52.477 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:52.477 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:52.477 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:52.477 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:52.477 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:52.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:39:52.494 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:39:52.495 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:39:52.495 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:39:52.495 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:39:52.495 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:52.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:39:52.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:39:52.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:39:52.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:39:52.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:39:52.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:39:52.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:39:52.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:39:52.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:39:52.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:39:52.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:39:52.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:39:52.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:39:52.496 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:39:52.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a04c5e4d-baa1-452c-90f1-069c27dec46b 2026-02-21T11:39:52.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a04c5e4d-baa1-452c-90f1-069c27dec46b' 2026-02-21T11:39:52.497 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 a04c5e4d-baa1-452c-90f1-069c27dec46b 2026-02-21T11:39:52.497 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:39:52.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCImZlp8wcsHhAAlQdlmcMxgz0AWKvehYJXMw== 2026-02-21T11:39:52.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCImZlp8wcsHhAAlQdlmcMxgz0AWKvehYJXMw=="}' 2026-02-21T11:39:52.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a04c5e4d-baa1-452c-90f1-069c27dec46b -i td/ceph-helpers/0/new.json 2026-02-21T11:39:52.608 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:52.614 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:39:52.615 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCImZlp8wcsHhAAlQdlmcMxgz0AWKvehYJXMw== --osd-uuid a04c5e4d-baa1-452c-90f1-069c27dec46b 2026-02-21T11:39:52.631 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:52.630+0000 7f81e2042780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:52.633 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:52.631+0000 7f81e2042780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:52.634 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:52.632+0000 7f81e2042780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:52.634 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:52.632+0000 7f81e2042780 -1 bdev(0x3be8c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:39:52.634 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:52.632+0000 7f81e2042780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:39:53.120 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:39:53.120 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:39:53.120 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:39:53.121 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:39:53.121 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:39:53.224 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:39:53.225 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:39:53.225 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:39:53.226 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:39:53.226 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:39:53.226 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:39:53.238 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:53.236+0000 7f32be80e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:53.239 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:53.238+0000 7f32be80e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:53.240 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:53.239+0000 7f32be80e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:39:53.322 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:39:53.323 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:39:53.323 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:39:53.323 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:39:53.323 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:39:53.323 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:53.323 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:39:53.324 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:39:53.324 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:53.324 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:53.420 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:39:53.444 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:53.443+0000 7f32be80e780 -1 Falling back to public interface 2026-02-21T11:39:53.771 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:39:53.769+0000 7f32be80e780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:39:54.422 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:39:54.422 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:39:54.422 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:39:54.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:39:54.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:39:54.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:39:54.519 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1445934023,v1:127.0.0.1:6801/1445934023] [v2:127.0.0.1:6802/1445934023,v1:127.0.0.1:6803/1445934023] exists,up a04c5e4d-baa1-452c-90f1-069c27dec46b 2026-02-21T11:39:54.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:39:54.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:39:54.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:39:54.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: test_get_last_scrub_stamp: create_rbd_pool 2026-02-21T11:39:54.520 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:39:54.610 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:39:54.615 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:39:54.615 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:39:54.898 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:39:54.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:39:55.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:39:56.188 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: test_get_last_scrub_stamp: wait_for_clean 2026-02-21T11:39:56.188 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:39:56.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:39:56.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:39:56.189 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:39:56.189 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:39:56.189 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:39:56.189 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:39:56.189 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:39:56.189 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:39:56.272 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:39:56.272 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:39:56.272 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:39:56.272 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:39:56.272 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:39:56.273 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:39:56.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:39:56.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:39:56.496 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:39:56.496 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:39:56.562 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836483 2026-02-21T11:39:56.562 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836483 2026-02-21T11:39:56.562 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T11:39:56.562 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:39:56.562 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836483 2026-02-21T11:39:56.562 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:39:56.563 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:39:56.563 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836483 2026-02-21T11:39:56.563 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:39:56.564 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836483 2026-02-21T11:39:56.564 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T11:39:56.565 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836483 2026-02-21T11:39:56.565 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:39:56.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T11:39:56.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:39:57.793 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:39:57.793 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:39:58.026 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T11:39:58.026 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:39:59.027 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T11:39:59.028 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:39:59.258 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T11:39:59.259 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:39:59.259 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:39:59.259 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:39:59.571 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T11:39:59.572 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:39:59.572 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:39:59.572 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:39:59.572 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:39:59.572 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:39:59.572 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:39:59.572 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:39:59.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T11:39:59.795 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:39:59.795 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:39:59.795 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:00.109 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T11:40:00.109 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:40:00.110 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:40:00.110 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1532: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2026-02-21T11:40:00.110 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T11:40:00.110 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T11:40:00.110 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T11:40:00.110 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T11:40:00.334 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1532: test_get_last_scrub_stamp: stamp=2026-02-21T11:39:54.833574+0000 2026-02-21T11:40:00.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: test -n 2026-02-21T11:39:54.833574+0000 2026-02-21T11:40:00.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1534: test_get_last_scrub_stamp: teardown td/ceph-helpers 2026-02-21T11:40:00.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:00.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:00.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:00.335 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:00.335 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:00.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:00.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:00.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:00.441 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:00.441 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:00.442 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:00.442 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:00.443 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:00.444 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:00.444 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:00.445 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:00.445 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:00.445 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:00.446 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:00.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:00.447 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:00.447 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:00.451 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:00.451 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:00.451 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:00.451 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:00.453 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:00.453 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:00.453 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:40:00.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_get_not_primary td/ceph-helpers 2026-02-21T11:40:00.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: test_get_not_primary: local dir=td/ceph-helpers 2026-02-21T11:40:00.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1231: test_get_not_primary: setup td/ceph-helpers 2026-02-21T11:40:00.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:40:00.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:40:00.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:00.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:00.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:00.454 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:00.454 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:00.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:00.454 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:00.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:00.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:00.455 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:00.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:00.456 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:00.456 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:00.457 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:00.457 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:00.457 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:00.457 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:00.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:00.458 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:00.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:00.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:00.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:00.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:00.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:00.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:00.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:00.462 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:00.462 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:00.462 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:40:00.463 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:40:00.463 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:00.463 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:00.463 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:40:00.464 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:40:00.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:40:00.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:40:00.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:40:00.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2026-02-21T11:40:00.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:40:00.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:40:00.466 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:40:00.466 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:40:00.466 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:40:00.466 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2026-02-21T11:40:00.484 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:40:00.485 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:00.485 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:00.485 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:00.485 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:00.485 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:00.485 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:00.485 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-21T11:40:00.509 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:40:00.511 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:40:00.511 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:00.512 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:00.512 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:40:00.512 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:40:00.512 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:00.512 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:00.512 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:00.512 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:00.512 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:00.512 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:00.512 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:00.512 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:00.512 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:40:00.554 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:40:00.554 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:00.554 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:00.554 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:40:00.554 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:40:00.555 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:00.555 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:00.555 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:00.555 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:00.555 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:00.555 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:00.555 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:00.555 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:00.555 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:40:00.596 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: test_get_not_primary: run_mgr td/ceph-helpers x 2026-02-21T11:40:00.596 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:40:00.596 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:40:00.596 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:40:00.596 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:40:00.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:40:00.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:40:00.699 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:40:00.699 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:00.699 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:00.699 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:00.699 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:00.699 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:00.700 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:00.700 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:00.700 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:00.717 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: test_get_not_primary: run_osd td/ceph-helpers 0 2026-02-21T11:40:00.718 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:40:00.718 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:40:00.718 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:40:00.718 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:40:00.718 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:40:00.718 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:40:00.718 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:40:00.718 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:40:00.718 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:40:00.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:40:00.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:40:00.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:40:00.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:40:00.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:40:00.719 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:40:00.719 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:00.719 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:00.719 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:00.719 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:00.719 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:00.719 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:00.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:00.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:40:00.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:40:00.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:40:00.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:40:00.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:40:00.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:40:00.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:40:00.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:40:00.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:40:00.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:40:00.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:40:00.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:40:00.721 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:40:00.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=60085373-b128-4a6a-b039-44a76d96ee31 2026-02-21T11:40:00.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 60085373-b128-4a6a-b039-44a76d96ee31' 2026-02-21T11:40:00.721 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 60085373-b128-4a6a-b039-44a76d96ee31 2026-02-21T11:40:00.721 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:40:00.728 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCQmZlpk41LKxAAUEbUFAvYxDHu1vjzdP3hAQ== 2026-02-21T11:40:00.728 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCQmZlpk41LKxAAUEbUFAvYxDHu1vjzdP3hAQ=="}' 2026-02-21T11:40:00.728 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 60085373-b128-4a6a-b039-44a76d96ee31 -i td/ceph-helpers/0/new.json 2026-02-21T11:40:00.831 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:00.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:40:00.839 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCQmZlpk41LKxAAUEbUFAvYxDHu1vjzdP3hAQ== --osd-uuid 60085373-b128-4a6a-b039-44a76d96ee31 2026-02-21T11:40:00.855 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:00.854+0000 7f283401b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:00.857 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:00.855+0000 7f283401b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:00.858 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:00.856+0000 7f283401b780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:00.858 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:00.856+0000 7f283401b780 -1 bdev(0x2544c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:40:00.858 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:00.856+0000 7f283401b780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:40:01.302 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:40:01.302 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:40:01.303 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:40:01.303 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:40:01.303 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:40:01.405 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:40:01.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:40:01.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:40:01.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:40:01.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:40:01.406 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:40:01.420 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:01.418+0000 7f49767d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:01.421 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:01.420+0000 7f49767d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:01.422 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:01.421+0000 7f49767d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:01.505 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:40:01.505 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:40:01.505 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:40:01.505 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:40:01.506 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:40:01.506 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:01.506 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:40:01.506 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:01.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:01.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:01.601 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:40:01.628 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:01.627+0000 7f49767d8780 -1 Falling back to public interface 2026-02-21T11:40:02.051 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:02.049+0000 7f49767d8780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:40:02.602 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:40:02.603 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:02.603 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:40:02.604 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:40:02.604 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:02.604 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:02.700 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3102837746,v1:127.0.0.1:6801/3102837746] [v2:127.0.0.1:6802/3102837746,v1:127.0.0.1:6803/3102837746] exists,up 60085373-b128-4a6a-b039-44a76d96ee31 2026-02-21T11:40:02.704 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:40:02.704 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:40:02.704 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:40:02.704 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: test_get_not_primary: run_osd td/ceph-helpers 1 2026-02-21T11:40:02.704 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:40:02.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:40:02.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T11:40:02.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:40:02.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-21T11:40:02.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:40:02.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:40:02.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:40:02.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:40:02.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-21T11:40:02.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-21T11:40:02.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:40:02.706 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:40:02.706 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:40:02.706 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:40:02.706 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:02.706 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:02.706 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:02.706 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:02.706 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:02.706 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:02.706 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:02.706 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:40:02.706 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:40:02.706 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:40:02.706 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:40:02.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:40:02.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:40:02.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:40:02.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:40:02.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:40:02.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:40:02.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:40:02.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-21T11:40:02.707 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:40:02.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9e5b5cdd-3c75-4231-88e4-9452257b158b 2026-02-21T11:40:02.707 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 9e5b5cdd-3c75-4231-88e4-9452257b158b' 2026-02-21T11:40:02.708 INFO:tasks.workunit.client.0.trial189.stdout:add osd1 9e5b5cdd-3c75-4231-88e4-9452257b158b 2026-02-21T11:40:02.708 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:40:02.713 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCSmZlpHtBwKhAAffsrm1uf3i1U1C35fmARSg== 2026-02-21T11:40:02.713 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCSmZlpHtBwKhAAffsrm1uf3i1U1C35fmARSg=="}' 2026-02-21T11:40:02.713 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9e5b5cdd-3c75-4231-88e4-9452257b158b -i td/ceph-helpers/1/new.json 2026-02-21T11:40:02.825 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:40:02.831 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-21T11:40:02.831 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCSmZlpHtBwKhAAffsrm1uf3i1U1C35fmARSg== --osd-uuid 9e5b5cdd-3c75-4231-88e4-9452257b158b 2026-02-21T11:40:02.847 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:02.846+0000 7fbfaa24a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:02.849 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:02.847+0000 7fbfaa24a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:02.850 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:02.848+0000 7fbfaa24a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:02.850 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:02.848+0000 7fbfaa24a780 -1 bdev(0x2512c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-21T11:40:02.850 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:02.848+0000 7fbfaa24a780 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-21T11:40:03.174 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-21T11:40:03.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:40:03.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T11:40:03.176 INFO:tasks.workunit.client.0.trial189.stdout:adding osd1 key to auth repository 2026-02-21T11:40:03.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:40:03.495 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T11:40:03.496 INFO:tasks.workunit.client.0.trial189.stdout:start osd.1 2026-02-21T11:40:03.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:40:03.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:40:03.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:40:03.497 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:40:03.510 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:03.509+0000 7f0b82e31780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:03.512 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:03.510+0000 7f0b82e31780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:03.512 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:03.511+0000 7f0b82e31780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:03.647 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:03.645+0000 7f0b82e31780 -1 Falling back to public interface 2026-02-21T11:40:03.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T11:40:03.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:40:03.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T11:40:03.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:40:03.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:40:03.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:03.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:40:03.722 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:03.722 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:03.722 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:40:03.951 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:40:03.962 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:03.960+0000 7f0b82e31780 -1 osd.1 0 log_to_monitors true 2026-02-21T11:40:04.952 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:40:04.952 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:04.952 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:40:04.953 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:40:04.953 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:04.953 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:40:05.178 INFO:tasks.workunit.client.0.trial189.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1666060613,v1:127.0.0.1:6811/1666060613] [v2:127.0.0.1:6812/1666060613,v1:127.0.0.1:6813/1666060613] exists,up 9e5b5cdd-3c75-4231-88e4-9452257b158b 2026-02-21T11:40:05.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:40:05.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:40:05.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:40:05.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: test_get_not_primary: create_rbd_pool 2026-02-21T11:40:05.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:40:05.398 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:40:05.405 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:40:05.405 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:40:05.652 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:40:05.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:40:06.660 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:40:06.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: test_get_not_primary: wait_for_clean 2026-02-21T11:40:06.942 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:40:06.942 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:40:06.942 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:40:06.942 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:40:06.942 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:40:06.942 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:40:06.942 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:40:06.942 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:40:06.942 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:40:07.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:40:07.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:40:07.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:40:07.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:40:07.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:40:07.025 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:40:07.250 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:40:07.250 INFO:tasks.workunit.client.0.trial189.stderr:1' 2026-02-21T11:40:07.250 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:40:07.250 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:40:07.250 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:40:07.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836483 2026-02-21T11:40:07.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836483 2026-02-21T11:40:07.315 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T11:40:07.316 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:40:07.316 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:40:07.381 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=38654705667 2026-02-21T11:40:07.381 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 38654705667 2026-02-21T11:40:07.381 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836483 1-38654705667' 2026-02-21T11:40:07.381 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:40:07.381 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836483 2026-02-21T11:40:07.382 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:40:07.382 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:40:07.382 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836483 2026-02-21T11:40:07.383 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:40:07.383 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836483 2026-02-21T11:40:07.384 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T11:40:07.384 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836483 2026-02-21T11:40:07.384 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:40:07.608 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T11:40:07.608 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:40:08.609 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:40:08.609 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:40:08.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-21T11:40:08.836 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:40:08.836 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-38654705667 2026-02-21T11:40:08.836 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:40:08.837 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:40:08.837 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-38654705667 2026-02-21T11:40:08.837 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:40:08.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=38654705667 2026-02-21T11:40:08.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-21T11:40:08.839 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 38654705667 2026-02-21T11:40:08.839 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:40:09.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-21T11:40:09.064 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:40:09.064 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:40:09.064 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:09.375 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T11:40:09.375 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:40:09.375 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:40:09.375 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:40:09.375 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:40:09.376 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:40:09.376 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:40:09.376 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:40:09.596 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T11:40:09.596 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:40:09.596 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:40:09.597 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:09.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T11:40:09.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:40:09.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:40:09.915 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: get_primary rbd GROUP 2026-02-21T11:40:09.915 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-21T11:40:09.915 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2026-02-21T11:40:09.915 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2026-02-21T11:40:09.915 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-21T11:40:10.142 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local primary=1 2026-02-21T11:40:10.142 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: test_get_not_primary: get_not_primary rbd GROUP 2026-02-21T11:40:10.143 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-21T11:40:10.143 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=GROUP 2026-02-21T11:40:10.143 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd GROUP 2026-02-21T11:40:10.143 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-21T11:40:10.143 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2026-02-21T11:40:10.143 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2026-02-21T11:40:10.143 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-21T11:40:10.382 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-21T11:40:10.382 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd GROUP 2026-02-21T11:40:10.382 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-21T11:40:10.610 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: test_get_not_primary: local not_primary=0 2026-02-21T11:40:10.610 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: test 0 '!=' 1 2026-02-21T11:40:10.610 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: test 0 = 0 -o 0 = 1 2026-02-21T11:40:10.610 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: teardown td/ceph-helpers 2026-02-21T11:40:10.610 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:10.610 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:10.610 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:10.610 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:10.610 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:10.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:10.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:10.611 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:10.717 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:10.718 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:10.718 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:10.719 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:10.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:10.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:10.721 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:10.722 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:10.722 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:10.722 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:10.722 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:10.723 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:10.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:10.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:10.729 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:10.729 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:10.729 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:10.729 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:10.731 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:10.731 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:10.731 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:40:10.732 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_get_num_active_clean td/ceph-helpers 2026-02-21T11:40:10.732 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: test_get_num_active_clean: local dir=td/ceph-helpers 2026-02-21T11:40:10.732 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: test_get_num_active_clean: setup td/ceph-helpers 2026-02-21T11:40:10.732 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:40:10.732 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:40:10.732 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:10.732 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:10.732 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:10.732 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:10.732 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:10.732 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:10.732 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:10.733 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:10.733 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:10.733 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:10.734 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:10.734 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:10.735 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:10.735 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:10.735 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:10.735 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:10.736 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:10.736 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:10.736 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:10.737 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:10.737 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:10.738 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:10.739 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:10.740 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:10.740 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:10.740 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:10.741 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:10.741 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:10.741 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:40:10.741 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:40:10.741 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:10.741 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:10.741 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:40:10.743 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:40:10.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:40:10.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:40:10.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:40:10.743 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:10.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:40:10.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:40:10.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:40:10.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:40:10.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:40:10.744 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:10.763 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:40:10.763 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:10.763 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:10.763 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:10.763 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:10.763 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:10.763 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:10.763 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:10.786 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:40:10.788 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:40:10.788 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:10.788 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:10.788 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:40:10.788 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:40:10.788 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:10.788 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:10.788 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:10.789 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:10.789 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:10.789 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:10.790 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:10.790 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:10.790 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:40:10.832 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:40:10.833 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:10.833 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:10.833 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:40:10.833 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:40:10.833 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:10.833 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:10.833 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:10.833 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:10.833 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:10.833 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:10.834 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:10.834 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:10.834 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:40:10.875 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: test_get_num_active_clean: run_mgr td/ceph-helpers x 2026-02-21T11:40:10.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:40:10.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:40:10.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:40:10.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:40:10.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:40:10.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:40:10.972 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:40:10.972 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:10.972 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:10.972 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:10.972 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:10.973 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:10.973 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:10.973 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:10.973 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:10.991 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: test_get_num_active_clean: run_osd td/ceph-helpers 0 2026-02-21T11:40:10.991 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:40:10.991 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:40:10.991 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:40:10.991 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:40:10.991 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:40:10.991 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:40:10.991 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:40:10.991 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:40:10.992 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:40:10.992 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:40:10.992 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:40:10.992 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:40:10.992 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:40:10.992 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:40:10.992 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:40:10.992 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:10.992 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:10.992 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:10.992 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:10.992 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:10.992 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:10.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:10.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:40:10.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:40:10.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:40:10.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:40:10.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:40:10.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:40:10.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:40:10.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:40:10.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:40:10.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:40:10.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:40:10.994 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:40:10.994 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:40:10.994 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=c0f22b74-7cb3-4809-b734-71744912a941 2026-02-21T11:40:10.994 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 c0f22b74-7cb3-4809-b734-71744912a941 2026-02-21T11:40:10.995 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 c0f22b74-7cb3-4809-b734-71744912a941' 2026-02-21T11:40:10.995 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:40:11.001 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCbmZlpPO8CABAAXZBsU30OPfOKNCWkyWlVcw== 2026-02-21T11:40:11.001 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCbmZlpPO8CABAAXZBsU30OPfOKNCWkyWlVcw=="}' 2026-02-21T11:40:11.001 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new c0f22b74-7cb3-4809-b734-71744912a941 -i td/ceph-helpers/0/new.json 2026-02-21T11:40:11.105 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:11.111 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:40:11.112 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCbmZlpPO8CABAAXZBsU30OPfOKNCWkyWlVcw== --osd-uuid c0f22b74-7cb3-4809-b734-71744912a941 2026-02-21T11:40:11.128 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:11.127+0000 7f9f2e5d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:11.130 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:11.128+0000 7f9f2e5d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:11.131 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:11.129+0000 7f9f2e5d4780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:11.131 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:11.129+0000 7f9f2e5d4780 -1 bdev(0x38eac00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:40:11.131 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:11.129+0000 7f9f2e5d4780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:40:11.567 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:40:11.567 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:40:11.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:40:11.568 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:40:11.569 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:40:11.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:40:11.673 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:40:11.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:40:11.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:40:11.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:40:11.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:40:11.687 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:11.686+0000 7f6d79866780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:11.689 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:11.687+0000 7f6d79866780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:11.690 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:11.688+0000 7f6d79866780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:11.776 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:40:11.777 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:40:11.777 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:40:11.777 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:40:11.777 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:40:11.777 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:11.778 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:11.778 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:40:11.778 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:11.778 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:11.873 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:40:11.912 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:11.910+0000 7f6d79866780 -1 Falling back to public interface 2026-02-21T11:40:12.209 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:12.207+0000 7f6d79866780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:40:12.874 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:40:12.875 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:12.876 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:40:12.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:40:12.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:12.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:12.973 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1964491660,v1:127.0.0.1:6801/1964491660] [v2:127.0.0.1:6802/1964491660,v1:127.0.0.1:6803/1964491660] exists,up c0f22b74-7cb3-4809-b734-71744912a941 2026-02-21T11:40:12.974 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:40:12.974 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:40:12.974 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:40:12.974 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1369: test_get_num_active_clean: create_rbd_pool 2026-02-21T11:40:12.974 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:40:13.067 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:40:13.073 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:40:13.073 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:40:13.346 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:40:13.353 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:40:14.355 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:40:14.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1370: test_get_num_active_clean: wait_for_clean 2026-02-21T11:40:14.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:40:14.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:40:14.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:40:14.637 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:40:14.637 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:40:14.637 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:40:14.637 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:40:14.637 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:40:14.637 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:40:14.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:40:14.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:40:14.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:40:14.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:40:14.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:40:14.721 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:40:14.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:40:14.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:40:14.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:40:14.945 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:40:15.010 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836483 2026-02-21T11:40:15.010 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836483 2026-02-21T11:40:15.010 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T11:40:15.010 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:40:15.010 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836483 2026-02-21T11:40:15.010 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:40:15.011 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:40:15.012 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836483 2026-02-21T11:40:15.012 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:40:15.013 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836483 2026-02-21T11:40:15.013 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T11:40:15.013 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836483 2026-02-21T11:40:15.013 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:40:15.241 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T11:40:15.241 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:40:16.242 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:40:16.242 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:40:16.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T11:40:16.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:40:17.469 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T11:40:17.470 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:40:17.701 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T11:40:17.702 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:40:17.702 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:40:17.702 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:18.019 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T11:40:18.019 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:40:18.020 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:40:18.020 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:40:18.020 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:40:18.020 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:40:18.020 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:40:18.020 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:40:18.243 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T11:40:18.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:40:18.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:40:18.243 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:18.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T11:40:18.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:40:18.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:40:18.556 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1371: test_get_num_active_clean: get_num_active_clean 2026-02-21T11:40:18.556 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:40:18.556 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:40:18.556 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:40:18.556 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:40:18.557 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:40:18.781 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1371: test_get_num_active_clean: local num_active_clean=4 2026-02-21T11:40:18.781 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: test 4 = 4 2026-02-21T11:40:18.782 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1373: test_get_num_active_clean: teardown td/ceph-helpers 2026-02-21T11:40:18.782 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:18.782 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:18.782 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:18.782 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:18.782 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:18.782 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:18.782 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:18.782 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:18.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:18.888 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:18.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:18.889 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:18.890 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:18.890 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:18.890 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:18.891 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:18.891 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:18.892 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:18.892 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:18.893 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:18.894 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:18.894 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:18.897 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:18.897 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:18.898 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:18.898 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_get_num_active_or_peered td/ceph-helpers 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1393: test_get_num_active_or_peered: local dir=td/ceph-helpers 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: test_get_num_active_or_peered: setup td/ceph-helpers 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:18.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:18.901 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:18.901 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:18.902 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:18.902 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:18.903 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:18.903 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:18.903 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:18.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:18.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:18.904 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:18.905 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:18.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:18.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:18.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:18.908 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:18.908 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:18.908 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:18.908 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:18.909 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:18.909 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:18.909 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:40:18.910 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:40:18.910 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:18.910 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:18.910 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:40:18.911 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:40:18.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:40:18.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:40:18.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:40:18.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:18.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:40:18.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:40:18.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:40:18.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:40:18.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:40:18.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:18.930 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:40:18.930 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:18.931 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:18.931 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:18.931 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:18.931 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:18.931 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:18.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:18.956 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:40:18.957 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:40:18.957 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:18.957 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:18.957 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:40:18.957 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:40:18.957 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:18.958 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:18.958 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:18.958 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:18.959 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:18.959 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:18.959 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:18.959 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:18.959 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:40:19.003 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:40:19.003 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:19.003 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:19.003 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:40:19.003 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:40:19.003 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:19.003 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:19.003 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:19.003 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:19.004 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:19.004 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:19.004 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:19.004 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:19.004 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:40:19.046 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2026-02-21T11:40:19.046 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:40:19.046 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:40:19.046 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:40:19.046 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:40:19.046 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:40:19.046 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:40:19.148 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:40:19.148 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:19.148 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:19.148 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:19.148 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:19.149 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:19.149 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:19.149 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:19.149 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:19.167 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2026-02-21T11:40:19.167 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:40:19.167 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:40:19.167 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:40:19.167 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:40:19.167 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:40:19.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:40:19.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:40:19.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:40:19.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:40:19.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:40:19.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:40:19.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:40:19.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:40:19.168 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:40:19.168 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:40:19.168 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:19.169 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:19.169 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:19.169 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:19.169 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:19.169 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:19.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:19.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:40:19.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:40:19.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:40:19.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:40:19.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:40:19.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:40:19.169 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:40:19.170 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:40:19.170 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:40:19.170 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:40:19.170 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:40:19.170 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:40:19.170 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:40:19.170 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1be2a2db-73d9-47fe-ba12-f13b3b33e971 2026-02-21T11:40:19.171 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 1be2a2db-73d9-47fe-ba12-f13b3b33e971 2026-02-21T11:40:19.171 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1be2a2db-73d9-47fe-ba12-f13b3b33e971' 2026-02-21T11:40:19.171 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:40:19.178 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCjmZlpcOKLChAA+IugWT3N2qfnH6HI6pXQQg== 2026-02-21T11:40:19.178 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCjmZlpcOKLChAA+IugWT3N2qfnH6HI6pXQQg=="}' 2026-02-21T11:40:19.178 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1be2a2db-73d9-47fe-ba12-f13b3b33e971 -i td/ceph-helpers/0/new.json 2026-02-21T11:40:19.279 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:19.286 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:40:19.286 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCjmZlpcOKLChAA+IugWT3N2qfnH6HI6pXQQg== --osd-uuid 1be2a2db-73d9-47fe-ba12-f13b3b33e971 2026-02-21T11:40:19.302 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:19.300+0000 7fec3f80f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:19.303 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:19.301+0000 7fec3f80f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:19.304 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:19.302+0000 7fec3f80f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:19.305 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:19.303+0000 7fec3f80f780 -1 bdev(0x31f4c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:40:19.305 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:19.303+0000 7fec3f80f780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:40:19.687 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:40:19.687 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:40:19.687 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:40:19.688 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:40:19.688 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:40:19.790 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:40:19.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:40:19.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:40:19.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:40:19.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:40:19.791 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:40:19.805 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:19.803+0000 7f7b9d01f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:19.807 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:19.805+0000 7f7b9d01f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:19.808 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:19.806+0000 7f7b9d01f780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:19.890 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:40:19.890 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:40:19.890 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:40:19.891 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:40:19.891 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:40:19.891 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:19.891 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:19.892 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:40:19.892 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:19.892 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:19.987 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:40:20.021 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:20.020+0000 7f7b9d01f780 -1 Falling back to public interface 2026-02-21T11:40:20.353 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:20.351+0000 7f7b9d01f780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:40:20.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:40:20.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:20.989 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:40:20.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:40:20.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:20.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:21.086 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3089180827,v1:127.0.0.1:6801/3089180827] [v2:127.0.0.1:6802/3089180827,v1:127.0.0.1:6803/3089180827] exists,up 1be2a2db-73d9-47fe-ba12-f13b3b33e971 2026-02-21T11:40:21.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:40:21.086 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:40:21.087 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:40:21.087 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1399: test_get_num_active_or_peered: create_rbd_pool 2026-02-21T11:40:21.087 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:40:21.185 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:40:21.191 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:40:21.191 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:40:21.382 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:40:21.388 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:40:22.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:40:22.667 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1400: test_get_num_active_or_peered: wait_for_clean 2026-02-21T11:40:22.667 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:40:22.667 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:40:22.667 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:40:22.667 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:40:22.668 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:40:22.668 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:40:22.668 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:40:22.668 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:40:22.668 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:40:22.751 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:40:22.752 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:40:22.752 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:40:22.752 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:40:22.752 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:40:22.752 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:40:22.976 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:40:22.977 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:40:22.977 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:40:22.977 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:40:23.040 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836483 2026-02-21T11:40:23.041 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836483 2026-02-21T11:40:23.041 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T11:40:23.041 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:40:23.041 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836483 2026-02-21T11:40:23.041 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:40:23.042 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:40:23.042 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836483 2026-02-21T11:40:23.042 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:40:23.043 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836483 2026-02-21T11:40:23.043 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T11:40:23.043 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836483 2026-02-21T11:40:23.043 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:40:23.269 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T11:40:23.270 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:40:23.270 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:40:23.270 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:23.582 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T11:40:23.582 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:40:23.582 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:40:23.582 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:40:23.582 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:40:23.582 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:40:23.582 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:40:23.582 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:40:23.805 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T11:40:23.805 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:40:23.805 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:40:23.805 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:24.115 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T11:40:24.116 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:40:24.116 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:40:24.116 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1401: test_get_num_active_or_peered: get_num_active_or_peered 2026-02-21T11:40:24.116 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1385: get_num_active_or_peered: local expression 2026-02-21T11:40:24.116 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1386: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2026-02-21T11:40:24.116 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1387: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2026-02-21T11:40:24.116 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1388: get_num_active_or_peered: ceph --format json pg dump pgs 2026-02-21T11:40:24.116 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1389: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2026-02-21T11:40:24.337 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1401: test_get_num_active_or_peered: local num_peered=4 2026-02-21T11:40:24.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: test 4 = 4 2026-02-21T11:40:24.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: test_get_num_active_or_peered: teardown td/ceph-helpers 2026-02-21T11:40:24.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:24.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:24.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:24.338 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:24.338 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:24.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:24.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:24.339 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:24.443 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:24.444 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:24.444 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:24.445 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:24.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:24.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:24.447 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:24.447 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:24.447 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:24.448 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:24.448 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:24.448 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:24.449 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:24.449 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:24.453 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:24.453 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:24.453 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:24.453 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:24.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:24.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:24.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:40:24.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_get_num_pgs td/ceph-helpers 2026-02-21T11:40:24.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1420: test_get_num_pgs: local dir=td/ceph-helpers 2026-02-21T11:40:24.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1422: test_get_num_pgs: setup td/ceph-helpers 2026-02-21T11:40:24.456 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:40:24.456 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:40:24.456 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:24.456 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:24.456 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:24.456 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:24.456 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:24.456 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:24.456 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:24.456 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:24.457 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:24.457 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:24.457 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:24.457 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:24.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:24.458 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:24.458 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:24.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:24.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:24.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:24.460 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:24.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:24.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:24.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:24.462 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:24.462 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:24.463 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:24.463 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:24.463 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:24.463 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:24.463 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:40:24.464 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:40:24.465 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:24.465 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:24.465 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:40:24.465 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:40:24.467 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:40:24.467 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:40:24.467 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:40:24.467 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:24.467 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:40:24.467 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:40:24.467 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:40:24.467 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:40:24.467 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:40:24.468 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:24.485 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:40:24.485 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:24.486 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:24.486 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:24.486 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:24.486 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:24.486 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:24.486 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:24.510 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:40:24.512 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:40:24.512 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:24.512 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:24.512 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:40:24.512 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:40:24.512 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:24.512 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:24.512 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:24.512 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:24.512 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:24.512 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:24.513 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:24.513 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:24.513 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:40:24.554 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:40:24.555 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:24.555 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:24.555 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:40:24.555 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:40:24.555 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:24.556 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:24.556 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:24.556 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:24.556 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:24.556 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:24.556 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:24.556 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:24.556 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:40:24.596 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1424: test_get_num_pgs: run_mgr td/ceph-helpers x 2026-02-21T11:40:24.596 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:40:24.596 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:40:24.596 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:40:24.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:40:24.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:40:24.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:40:24.700 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:40:24.700 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:24.700 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:24.700 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:24.700 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:24.700 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:24.700 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:24.701 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:24.701 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:24.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: test_get_num_pgs: run_osd td/ceph-helpers 0 2026-02-21T11:40:24.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:40:24.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:40:24.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:40:24.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:40:24.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:40:24.719 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:40:24.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:40:24.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:40:24.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:40:24.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:40:24.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:40:24.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:40:24.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:40:24.720 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:40:24.720 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:40:24.720 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:24.720 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:24.720 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:24.720 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:24.721 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:24.721 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:24.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:24.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:40:24.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:40:24.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:40:24.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:40:24.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:40:24.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:40:24.721 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:40:24.722 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:40:24.722 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:40:24.722 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:40:24.722 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:40:24.722 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:40:24.722 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:40:24.722 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0c1ba1b3-7ebf-4cd3-a879-2ae94d50b1f3 2026-02-21T11:40:24.722 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 0c1ba1b3-7ebf-4cd3-a879-2ae94d50b1f3 2026-02-21T11:40:24.723 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0c1ba1b3-7ebf-4cd3-a879-2ae94d50b1f3' 2026-02-21T11:40:24.723 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:40:24.729 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQComZlpukJfKxAAFTo290L0YH+52lM7Xx+3Rw== 2026-02-21T11:40:24.729 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQComZlpukJfKxAAFTo290L0YH+52lM7Xx+3Rw=="}' 2026-02-21T11:40:24.729 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0c1ba1b3-7ebf-4cd3-a879-2ae94d50b1f3 -i td/ceph-helpers/0/new.json 2026-02-21T11:40:24.831 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:24.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:40:24.838 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQComZlpukJfKxAAFTo290L0YH+52lM7Xx+3Rw== --osd-uuid 0c1ba1b3-7ebf-4cd3-a879-2ae94d50b1f3 2026-02-21T11:40:24.855 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:24.853+0000 7fa47720a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:24.857 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:24.855+0000 7fa47720a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:24.857 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:24.856+0000 7fa47720a780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:24.858 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:24.856+0000 7fa47720a780 -1 bdev(0x42aac00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:40:24.858 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:24.856+0000 7fa47720a780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:40:25.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:40:25.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:40:25.274 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:40:25.275 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:40:25.275 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:40:25.376 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:40:25.377 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:40:25.377 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:40:25.378 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:40:25.378 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:40:25.378 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:40:25.391 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:25.389+0000 7f1ebd642780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:25.392 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:25.391+0000 7f1ebd642780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:25.393 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:25.391+0000 7f1ebd642780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:25.474 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:40:25.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:40:25.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:40:25.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:40:25.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:40:25.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:25.475 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:40:25.475 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:25.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:25.476 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:25.574 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:40:25.619 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:25.617+0000 7f1ebd642780 -1 Falling back to public interface 2026-02-21T11:40:25.991 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:25.989+0000 7f1ebd642780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:40:26.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:40:26.575 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:26.576 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:40:26.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:40:26.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:26.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:26.672 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3767979895,v1:127.0.0.1:6801/3767979895] [v2:127.0.0.1:6802/3767979895,v1:127.0.0.1:6803/3767979895] exists,up 0c1ba1b3-7ebf-4cd3-a879-2ae94d50b1f3 2026-02-21T11:40:26.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:40:26.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:40:26.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:40:26.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1426: test_get_num_pgs: create_rbd_pool 2026-02-21T11:40:26.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:40:26.769 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:40:26.775 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:40:26.775 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:40:26.951 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:40:26.957 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:40:27.959 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:40:28.237 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1427: test_get_num_pgs: wait_for_clean 2026-02-21T11:40:28.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:40:28.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:40:28.238 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:40:28.238 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:40:28.239 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:40:28.239 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:40:28.239 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:40:28.239 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:40:28.239 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:40:28.322 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:40:28.322 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:40:28.322 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:40:28.323 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:40:28.323 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:40:28.323 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:40:28.548 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:40:28.548 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:40:28.548 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:40:28.548 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:40:28.613 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836483 2026-02-21T11:40:28.613 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836483 2026-02-21T11:40:28.613 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T11:40:28.613 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:40:28.613 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836483 2026-02-21T11:40:28.614 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:40:28.614 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:40:28.614 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836483 2026-02-21T11:40:28.615 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:40:28.616 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836483 2026-02-21T11:40:28.616 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T11:40:28.616 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836483 2026-02-21T11:40:28.616 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:40:28.844 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T11:40:28.844 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:40:28.845 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:40:28.845 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:29.157 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T11:40:29.157 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:40:29.158 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:40:29.158 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:40:29.158 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:40:29.158 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:40:29.158 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:40:29.158 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:40:29.379 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T11:40:29.379 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:40:29.380 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:40:29.380 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:29.688 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T11:40:29.688 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:40:29.689 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:40:29.689 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1428: test_get_num_pgs: get_num_pgs 2026-02-21T11:40:29.689 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:40:29.689 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:30.003 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1428: test_get_num_pgs: local num_pgs=4 2026-02-21T11:40:30.003 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: test 4 -gt 0 2026-02-21T11:40:30.004 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1430: test_get_num_pgs: teardown td/ceph-helpers 2026-02-21T11:40:30.004 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:30.004 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:30.004 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:30.004 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:30.004 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:30.004 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:30.004 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:30.004 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:30.110 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:30.110 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:30.111 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:30.111 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:30.112 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:30.112 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:30.113 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:30.114 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:30.114 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:30.114 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:30.115 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:30.115 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:30.116 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:30.116 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:30.121 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:30.121 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:30.121 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:30.121 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:30.125 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:30.125 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:30.125 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:40:30.126 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_get_op_scheduler td/ceph-helpers 2026-02-21T11:40:30.126 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: test_get_op_scheduler: local dir=td/ceph-helpers 2026-02-21T11:40:30.126 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2346: test_get_op_scheduler: setup td/ceph-helpers 2026-02-21T11:40:30.126 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:40:30.126 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:40:30.126 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:30.126 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:30.127 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:30.127 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:30.127 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:30.127 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:30.127 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:30.127 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:30.127 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:30.128 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:30.128 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:30.128 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:30.131 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:30.131 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:30.131 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:30.131 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:30.131 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:30.131 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:30.132 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:30.132 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:30.132 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:30.132 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:30.132 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:30.132 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:30.133 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:30.133 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:30.136 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:30.136 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:30.136 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:40:30.136 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:40:30.136 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:30.137 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:30.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:40:30.137 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:40:30.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:40:30.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:40:30.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:40:30.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2348: test_get_op_scheduler: run_mon td/ceph-helpers a 2026-02-21T11:40:30.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:40:30.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:40:30.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:40:30.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:40:30.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:40:30.138 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-21T11:40:30.152 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:40:30.152 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:30.152 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:30.153 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:30.153 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:30.153 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:30.153 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:30.153 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-21T11:40:30.176 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:40:30.177 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:40:30.178 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:30.178 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:30.178 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:40:30.178 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:40:30.178 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:30.178 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:30.178 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:30.178 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:30.178 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:30.178 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:30.178 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:30.179 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:30.179 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:40:30.222 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:40:30.222 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:30.222 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:30.222 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:40:30.222 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:40:30.222 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:30.223 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:30.223 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:30.223 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:30.223 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:30.223 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:30.223 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:30.224 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:30.224 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:40:30.264 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2349: test_get_op_scheduler: run_mgr td/ceph-helpers x 2026-02-21T11:40:30.264 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:40:30.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:40:30.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:40:30.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:40:30.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:40:30.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:40:30.363 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:40:30.364 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:30.364 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:30.364 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:30.364 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:30.364 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:30.364 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:30.364 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:30.365 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:30.386 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2351: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2026-02-21T11:40:30.386 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:40:30.386 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:40:30.386 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:40:30.386 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:40:30.386 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:40:30.386 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:40:30.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:40:30.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:40:30.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:40:30.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:40:30.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:40:30.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:40:30.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:40:30.387 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:40:30.387 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:40:30.387 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:30.388 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:30.388 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:30.388 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:30.388 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:30.388 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:30.388 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:30.388 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:40:30.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:40:30.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:40:30.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:40:30.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:40:30.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:40:30.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:40:30.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:40:30.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:40:30.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:40:30.389 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=wpq 2026-02-21T11:40:30.390 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:40:30.390 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:40:30.390 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=616ca89e-b270-469c-9a20-5f2427e4c78b 2026-02-21T11:40:30.390 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 616ca89e-b270-469c-9a20-5f2427e4c78b' 2026-02-21T11:40:30.390 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 616ca89e-b270-469c-9a20-5f2427e4c78b 2026-02-21T11:40:30.391 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:40:30.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCumZlpmnlRFxAA0uq9e/hyE3H+F4m8iAXG8A== 2026-02-21T11:40:30.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCumZlpmnlRFxAA0uq9e/hyE3H+F4m8iAXG8A=="}' 2026-02-21T11:40:30.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 616ca89e-b270-469c-9a20-5f2427e4c78b -i td/ceph-helpers/0/new.json 2026-02-21T11:40:30.496 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:30.502 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:40:30.503 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQCumZlpmnlRFxAA0uq9e/hyE3H+F4m8iAXG8A== --osd-uuid 616ca89e-b270-469c-9a20-5f2427e4c78b 2026-02-21T11:40:30.519 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:30.517+0000 7f8589087780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:30.520 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:30.518+0000 7f8589087780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:30.521 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:30.519+0000 7f8589087780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:30.521 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:30.520+0000 7f8589087780 -1 bdev(0x3fecc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:40:30.522 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:30.520+0000 7f8589087780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:40:30.969 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:40:30.969 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:40:30.970 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:40:30.971 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:40:30.971 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:40:31.074 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:40:31.075 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:40:31.075 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2026-02-21T11:40:31.075 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:40:31.075 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:40:31.075 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:40:31.089 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:31.087+0000 7f2573fe3780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:31.090 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:31.089+0000 7f2573fe3780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:31.091 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:31.089+0000 7f2573fe3780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:31.173 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:40:31.173 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:40:31.173 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:40:31.173 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:40:31.173 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:40:31.173 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:31.174 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:31.174 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:40:31.174 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:31.175 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:31.269 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:40:31.300 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:31.299+0000 7f2573fe3780 -1 Falling back to public interface 2026-02-21T11:40:31.588 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:31.586+0000 7f2573fe3780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:40:32.271 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:40:32.271 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:32.272 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:40:32.272 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:40:32.272 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:32.273 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:32.368 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1241405633,v1:127.0.0.1:6801/1241405633] [v2:127.0.0.1:6802/1241405633,v1:127.0.0.1:6803/1241405633] exists,up 616ca89e-b270-469c-9a20-5f2427e4c78b 2026-02-21T11:40:32.369 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:40:32.369 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:40:32.369 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:40:32.369 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: test_get_op_scheduler: get_op_scheduler 0 2026-02-21T11:40:32.369 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2338: get_op_scheduler: local id=0 2026-02-21T11:40:32.369 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: get_op_scheduler: get_config osd 0 osd_op_queue 2026-02-21T11:40:32.369 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2026-02-21T11:40:32.370 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=0 2026-02-21T11:40:32.370 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_op_queue 2026-02-21T11:40:32.370 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_op_queue 2026-02-21T11:40:32.370 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.0 2026-02-21T11:40:32.370 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-21T11:40:32.370 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-21T11:40:32.370 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:32.370 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:32.370 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:32.370 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-osd.0.asok 2026-02-21T11:40:32.371 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:32.371 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-osd.0.asok config get osd_op_queue 2026-02-21T11:40:32.422 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: test_get_op_scheduler: test wpq = wpq 2026-02-21T11:40:32.422 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2026-02-21T11:40:32.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:40:32.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:40:32.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T11:40:32.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:40:32.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-21T11:40:32.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:40:32.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:40:32.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:40:32.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:40:32.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-21T11:40:32.424 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-21T11:40:32.424 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:40:32.424 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:40:32.424 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:40:32.424 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:40:32.424 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:32.424 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:32.424 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:32.424 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:32.424 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:32.424 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:32.424 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:32.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:40:32.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:40:32.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:40:32.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:40:32.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:40:32.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:40:32.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:40:32.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:40:32.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:40:32.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:40:32.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-21T11:40:32.425 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-21T11:40:32.425 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:40:32.426 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d744e2f8-66af-4868-865c-f33f041aa6ec 2026-02-21T11:40:32.426 INFO:tasks.workunit.client.0.trial189.stdout:add osd1 d744e2f8-66af-4868-865c-f33f041aa6ec 2026-02-21T11:40:32.427 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 d744e2f8-66af-4868-865c-f33f041aa6ec' 2026-02-21T11:40:32.427 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:40:32.432 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCwmZlpyKu4GRAAkPdF6jblI3NgxEtoxIPF+A== 2026-02-21T11:40:32.433 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCwmZlpyKu4GRAAkPdF6jblI3NgxEtoxIPF+A=="}' 2026-02-21T11:40:32.433 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d744e2f8-66af-4868-865c-f33f041aa6ec -i td/ceph-helpers/1/new.json 2026-02-21T11:40:32.534 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:40:32.539 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-21T11:40:32.540 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCwmZlpyKu4GRAAkPdF6jblI3NgxEtoxIPF+A== --osd-uuid d744e2f8-66af-4868-865c-f33f041aa6ec 2026-02-21T11:40:32.555 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:32.553+0000 7fdfe073d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:32.556 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:32.555+0000 7fdfe073d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:32.557 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:32.556+0000 7fdfe073d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:32.557 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:32.556+0000 7fdfe073d780 -1 bdev(0x2ba4c00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-21T11:40:32.558 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:32.556+0000 7fdfe073d780 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-21T11:40:32.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-21T11:40:32.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:40:32.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T11:40:32.889 INFO:tasks.workunit.client.0.trial189.stdout:adding osd1 key to auth repository 2026-02-21T11:40:32.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:40:33.206 INFO:tasks.workunit.client.0.trial189.stdout:start osd.1 2026-02-21T11:40:33.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T11:40:33.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-21T11:40:33.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:40:33.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:40:33.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:40:33.221 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:33.219+0000 7fa029044780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:33.222 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:33.221+0000 7fa029044780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:33.223 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:33.222+0000 7fa029044780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:33.368 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:33.366+0000 7fa029044780 -1 Falling back to public interface 2026-02-21T11:40:33.436 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T11:40:33.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:40:33.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T11:40:33.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:40:33.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:40:33.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:33.437 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:33.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:40:33.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:33.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:40:33.663 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:40:33.686 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:33.684+0000 7fa029044780 -1 osd.1 0 log_to_monitors true 2026-02-21T11:40:34.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:40:34.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:34.665 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:40:34.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:40:34.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:34.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:40:34.892 INFO:tasks.workunit.client.0.trial189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1012932087,v1:127.0.0.1:6811/1012932087] [v2:127.0.0.1:6812/1012932087,v1:127.0.0.1:6813/1012932087] exists,up d744e2f8-66af-4868-865c-f33f041aa6ec 2026-02-21T11:40:34.894 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:40:34.895 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:40:34.895 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:40:34.895 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: test_get_op_scheduler: get_op_scheduler 1 2026-02-21T11:40:34.895 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2338: get_op_scheduler: local id=1 2026-02-21T11:40:34.895 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2340: get_op_scheduler: get_config osd 1 osd_op_queue 2026-02-21T11:40:34.895 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2026-02-21T11:40:34.895 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=1 2026-02-21T11:40:34.895 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_op_queue 2026-02-21T11:40:34.895 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_op_queue 2026-02-21T11:40:34.895 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.1 2026-02-21T11:40:34.895 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-21T11:40:34.896 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-21T11:40:34.896 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:34.896 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:34.896 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:34.896 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-osd.1.asok 2026-02-21T11:40:34.896 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:34.896 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-osd.1.asok config get osd_op_queue 2026-02-21T11:40:34.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2026-02-21T11:40:34.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: test_get_op_scheduler: teardown td/ceph-helpers 2026-02-21T11:40:34.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:34.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:34.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:34.941 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:34.941 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:34.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:34.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:34.942 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:35.047 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:35.048 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:35.048 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:35.049 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:35.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:35.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:35.051 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:35.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:35.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:35.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:35.052 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:35.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:35.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:35.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:35.058 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:35.058 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:35.058 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:35.058 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:35.059 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:35.059 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:35.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:40:35.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2026-02-21T11:40:35.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1450: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2026-02-21T11:40:35.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1452: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2026-02-21T11:40:35.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:40:35.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:40:35.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:35.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:35.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:35.060 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:35.060 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:35.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:35.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:35.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:35.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:35.061 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:35.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:35.062 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:35.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:35.063 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:35.063 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:35.064 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:35.064 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:35.064 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:35.065 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:35.065 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:35.066 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:35.066 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:35.067 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:35.067 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:35.067 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:35.067 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:35.068 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:35.068 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:35.068 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:40:35.069 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:40:35.069 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:35.069 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:35.069 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:40:35.070 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:40:35.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:40:35.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:40:35.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:40:35.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1453: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:35.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:40:35.071 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:40:35.072 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:40:35.072 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:40:35.072 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:40:35.072 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:35.090 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:40:35.090 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:35.090 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:35.091 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:35.091 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:35.091 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:35.091 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:35.091 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:35.115 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:40:35.116 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:40:35.116 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:35.116 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:35.116 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:40:35.116 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:40:35.116 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:35.117 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:35.117 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:35.118 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:35.118 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:35.118 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:35.118 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:35.119 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:35.119 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:40:35.160 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:40:35.161 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:35.161 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:35.161 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:40:35.161 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:40:35.161 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:35.161 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:35.161 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:35.161 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:35.161 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:35.161 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:35.162 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:35.162 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:35.162 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:40:35.203 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1454: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2026-02-21T11:40:35.203 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:40:35.203 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:40:35.203 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:40:35.203 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:40:35.203 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:40:35.204 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:40:35.300 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:40:35.300 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:35.300 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:35.300 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:35.300 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:35.300 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:35.301 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:35.301 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:35.302 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:40:35.318 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:35.319 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:35.319 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:35.319 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:35.319 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:35.319 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:35.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:35.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:40:35.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:40:35.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:40:35.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:40:35.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:40:35.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:40:35.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:40:35.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:40:35.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:40:35.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:40:35.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:40:35.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:40:35.320 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:40:35.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ff265160-7bd8-4c90-b962-59c8b8ed1f28 2026-02-21T11:40:35.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ff265160-7bd8-4c90-b962-59c8b8ed1f28' 2026-02-21T11:40:35.320 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 ff265160-7bd8-4c90-b962-59c8b8ed1f28 2026-02-21T11:40:35.320 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:40:35.330 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCzmZlpjtGcExAA8okh9eCiaVhtBXRz6s2uXg== 2026-02-21T11:40:35.330 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCzmZlpjtGcExAA8okh9eCiaVhtBXRz6s2uXg=="}' 2026-02-21T11:40:35.330 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ff265160-7bd8-4c90-b962-59c8b8ed1f28 -i td/ceph-helpers/0/new.json 2026-02-21T11:40:35.433 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:35.439 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:40:35.440 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCzmZlpjtGcExAA8okh9eCiaVhtBXRz6s2uXg== --osd-uuid ff265160-7bd8-4c90-b962-59c8b8ed1f28 2026-02-21T11:40:35.455 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:35.454+0000 7fc58682e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:35.457 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:35.455+0000 7fc58682e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:35.458 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:35.456+0000 7fc58682e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:35.458 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:35.456+0000 7fc58682e780 -1 bdev(0x2d9cc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:40:35.458 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:35.457+0000 7fc58682e780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:40:35.878 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:40:35.878 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:40:35.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:40:35.879 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:40:35.879 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:40:35.982 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:40:35.983 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:40:35.984 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:40:35.984 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:40:35.984 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:40:35.984 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:40:35.997 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:35.995+0000 7fdf66a42780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:35.998 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:35.996+0000 7fdf66a42780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:35.999 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:35.997+0000 7fdf66a42780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:36.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:40:36.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:40:36.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:40:36.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:40:36.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:40:36.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:36.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:40:36.083 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:36.084 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:36.084 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:36.181 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:40:36.203 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:36.202+0000 7fdf66a42780 -1 Falling back to public interface 2026-02-21T11:40:36.625 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:36.623+0000 7fdf66a42780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:40:37.182 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:40:37.183 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:37.183 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:40:37.184 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:40:37.184 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:37.184 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:37.278 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2931938187,v1:127.0.0.1:6801/2931938187] [v2:127.0.0.1:6802/2931938187,v1:127.0.0.1:6803/2931938187] exists,up ff265160-7bd8-4c90-b962-59c8b8ed1f28 2026-02-21T11:40:37.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:40:37.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:40:37.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:40:37.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1456: test_get_osd_id_used_by_pgs: create_rbd_pool 2026-02-21T11:40:37.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:40:37.374 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:40:37.380 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:40:37.380 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:40:37.555 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:40:37.562 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:40:38.564 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:40:38.843 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1457: test_get_osd_id_used_by_pgs: wait_for_clean 2026-02-21T11:40:38.843 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:40:38.843 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:40:38.843 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:40:38.843 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:40:38.845 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:40:38.845 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:40:38.845 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:40:38.845 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:40:38.845 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:40:38.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:40:38.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:40:38.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:40:38.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:40:38.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:40:38.927 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:40:39.151 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:40:39.151 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:40:39.151 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:40:39.151 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:40:39.217 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836483 2026-02-21T11:40:39.217 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836483 2026-02-21T11:40:39.218 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T11:40:39.218 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:40:39.218 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836483 2026-02-21T11:40:39.218 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:40:39.219 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:40:39.219 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836483 2026-02-21T11:40:39.219 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:40:39.220 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836483 2026-02-21T11:40:39.220 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T11:40:39.220 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836483 2026-02-21T11:40:39.221 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:40:39.445 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T11:40:39.445 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:40:39.446 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:40:39.446 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:39.758 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T11:40:39.759 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:40:39.759 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:40:39.759 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:40:39.759 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:40:39.759 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:40:39.759 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:40:39.759 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:40:39.984 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T11:40:39.984 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:40:39.984 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:40:39.984 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:40.310 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T11:40:40.310 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:40:40.310 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:40:40.310 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2026-02-21T11:40:40.310 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: uniq 2026-02-21T11:40:40.310 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2026-02-21T11:40:40.311 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2026-02-21T11:40:40.311 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: sort 2026-02-21T11:40:40.533 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: local osd_ids=0 2026-02-21T11:40:40.533 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: test 0 = 0 2026-02-21T11:40:40.533 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1460: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2026-02-21T11:40:40.533 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:40.533 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:40.533 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:40.533 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:40.533 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:40.533 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:40.533 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:40.533 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:40.641 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:40.641 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:40.642 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:40.642 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:40.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:40.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:40.644 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:40.645 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:40.645 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:40.645 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:40.646 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:40.646 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:40.647 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:40.647 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:40.651 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:40.651 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:40.651 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:40.651 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:40.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:40.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:40.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:40:40.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_get_osds td/ceph-helpers 2026-02-21T11:40:40.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: test_get_osds: local dir=td/ceph-helpers 2026-02-21T11:40:40.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1024: test_get_osds: setup td/ceph-helpers 2026-02-21T11:40:40.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:40:40.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:40:40.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:40.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:40.652 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:40.652 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:40.653 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:40.653 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:40.653 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:40.653 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:40.654 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:40.655 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:40.655 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:40.656 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:40.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:40.657 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:40.657 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:40.657 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:40.657 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:40.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:40.658 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:40.658 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:40.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:40.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:40.661 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:40.661 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:40.661 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:40.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:40.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:40.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:40.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:40:40.663 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:40:40.663 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:40.663 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:40.663 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:40:40.665 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:40:40.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:40:40.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:40:40.665 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:40:40.666 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2026-02-21T11:40:40.666 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:40:40.666 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:40:40.666 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:40:40.666 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:40:40.666 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:40:40.666 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2026-02-21T11:40:40.684 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:40:40.684 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:40.684 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:40.684 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:40.684 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:40.684 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:40.684 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:40.684 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-21T11:40:40.708 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:40:40.709 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:40:40.709 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:40.709 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:40.710 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:40:40.710 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:40:40.710 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:40.710 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:40.710 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:40.710 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:40.710 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:40.710 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:40.710 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:40.711 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:40.711 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:40:40.751 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:40:40.751 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:40.751 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:40.752 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:40:40.752 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:40:40.753 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:40.753 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:40.753 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:40.753 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:40.753 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:40.753 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:40.754 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:40.754 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:40.754 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:40:40.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1026: test_get_osds: run_mgr td/ceph-helpers x 2026-02-21T11:40:40.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:40:40.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:40:40.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:40:40.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:40:40.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:40:40.794 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:40:40.889 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:40:40.889 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:40.889 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:40.889 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:40.889 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:40.889 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:40.889 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:40.889 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:40.890 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:40.905 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: test_get_osds: run_osd td/ceph-helpers 0 2026-02-21T11:40:40.905 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:40:40.905 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:40:40.905 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:40:40.905 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:40.906 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:40.907 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:40.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:40.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:40:40.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:40:40.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:40:40.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:40:40.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:40:40.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:40:40.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:40:40.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:40:40.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:40:40.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:40:40.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:40:40.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:40:40.907 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:40:40.908 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=74b961e4-aff0-4fa5-9240-448c26ea2063 2026-02-21T11:40:40.908 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 74b961e4-aff0-4fa5-9240-448c26ea2063' 2026-02-21T11:40:40.908 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 74b961e4-aff0-4fa5-9240-448c26ea2063 2026-02-21T11:40:40.908 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:40:40.917 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC4mZlp94mhNhAAXxnJTaayaKoAPfBuL6JFmA== 2026-02-21T11:40:40.917 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC4mZlp94mhNhAAXxnJTaayaKoAPfBuL6JFmA=="}' 2026-02-21T11:40:40.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 74b961e4-aff0-4fa5-9240-448c26ea2063 -i td/ceph-helpers/0/new.json 2026-02-21T11:40:41.019 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:41.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:40:41.025 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4mZlp94mhNhAAXxnJTaayaKoAPfBuL6JFmA== --osd-uuid 74b961e4-aff0-4fa5-9240-448c26ea2063 2026-02-21T11:40:41.041 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:41.039+0000 7f69e0687780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:41.042 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:41.041+0000 7f69e0687780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:41.043 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:41.041+0000 7f69e0687780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:41.044 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:41.042+0000 7f69e0687780 -1 bdev(0x3976c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:40:41.044 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:41.042+0000 7f69e0687780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:40:41.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:40:41.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:40:41.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:40:41.479 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:40:41.479 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:40:41.585 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:40:41.585 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:40:41.586 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:40:41.586 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:40:41.586 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:40:41.586 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:40:41.600 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:41.598+0000 7f8e50a39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:41.601 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:41.599+0000 7f8e50a39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:41.602 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:41.600+0000 7f8e50a39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:41.684 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:40:41.684 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:40:41.684 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:40:41.685 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:40:41.685 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:40:41.685 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:41.685 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:41.686 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:40:41.686 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:41.686 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:41.782 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:40:41.839 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:41.837+0000 7f8e50a39780 -1 Falling back to public interface 2026-02-21T11:40:42.192 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:42.190+0000 7f8e50a39780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:40:42.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:40:42.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:42.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:40:42.783 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:40:42.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:42.783 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:42.882 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1631243038,v1:127.0.0.1:6801/1631243038] [v2:127.0.0.1:6802/1631243038,v1:127.0.0.1:6803/1631243038] exists,up 74b961e4-aff0-4fa5-9240-448c26ea2063 2026-02-21T11:40:42.886 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:40:42.886 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:40:42.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:40:42.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1028: test_get_osds: run_osd td/ceph-helpers 1 2026-02-21T11:40:42.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:40:42.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:40:42.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-21T11:40:42.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:40:42.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-21T11:40:42.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:40:42.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:40:42.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:40:42.887 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:40:42.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-21T11:40:42.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-21T11:40:42.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:40:42.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:40:42.888 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:40:42.888 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:40:42.888 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:42.888 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:42.888 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:42.888 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:42.888 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:42.888 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:42.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:42.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:40:42.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:40:42.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:40:42.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:40:42.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:40:42.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:40:42.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:40:42.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:40:42.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:40:42.889 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:40:42.890 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:40:42.890 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-21T11:40:42.890 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:40:42.890 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=33ce31bb-096e-4675-9dcb-8aeebf1f5c78 2026-02-21T11:40:42.890 INFO:tasks.workunit.client.0.trial189.stdout:add osd1 33ce31bb-096e-4675-9dcb-8aeebf1f5c78 2026-02-21T11:40:42.891 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 33ce31bb-096e-4675-9dcb-8aeebf1f5c78' 2026-02-21T11:40:42.891 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:40:42.896 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC6mZlpec9cNRAAIIfImHOO+J8bILoerV+GeQ== 2026-02-21T11:40:42.897 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC6mZlpec9cNRAAIIfImHOO+J8bILoerV+GeQ=="}' 2026-02-21T11:40:42.897 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 33ce31bb-096e-4675-9dcb-8aeebf1f5c78 -i td/ceph-helpers/1/new.json 2026-02-21T11:40:42.999 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:40:43.005 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-21T11:40:43.006 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6mZlpec9cNRAAIIfImHOO+J8bILoerV+GeQ== --osd-uuid 33ce31bb-096e-4675-9dcb-8aeebf1f5c78 2026-02-21T11:40:43.021 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:43.019+0000 7f0dc6e39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:43.023 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:43.021+0000 7f0dc6e39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:43.023 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:43.022+0000 7f0dc6e39780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:43.024 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:43.022+0000 7f0dc6e39780 -1 bdev(0x35bcc00 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-21T11:40:43.024 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:43.022+0000 7f0dc6e39780 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-21T11:40:43.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-21T11:40:43.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:40:43.344 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-21T11:40:43.344 INFO:tasks.workunit.client.0.trial189.stdout:adding osd1 key to auth repository 2026-02-21T11:40:43.344 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:40:43.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-21T11:40:43.674 INFO:tasks.workunit.client.0.trial189.stdout:start osd.1 2026-02-21T11:40:43.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:40:43.675 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:40:43.675 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:40:43.675 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:40:43.688 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:43.686+0000 7fe403a42780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:43.690 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:43.688+0000 7fe403a42780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:43.691 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:43.689+0000 7fe403a42780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:43.821 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:43.820+0000 7fe403a42780 -1 Falling back to public interface 2026-02-21T11:40:43.898 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-21T11:40:43.898 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:40:43.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-21T11:40:43.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:40:43.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:40:43.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:43.899 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:40:43.899 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:43.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:43.900 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:40:44.112 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:44.111+0000 7fe403a42780 -1 osd.1 0 log_to_monitors true 2026-02-21T11:40:44.125 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:40:45.126 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:40:45.127 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:45.127 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:40:45.128 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:40:45.128 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:45.128 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-21T11:40:45.351 INFO:tasks.workunit.client.0.trial189.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/470919935,v1:127.0.0.1:6811/470919935] [v2:127.0.0.1:6812/470919935,v1:127.0.0.1:6813/470919935] exists,up 33ce31bb-096e-4675-9dcb-8aeebf1f5c78 2026-02-21T11:40:45.352 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:40:45.352 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:40:45.352 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:40:45.352 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: test_get_osds: create_rbd_pool 2026-02-21T11:40:45.352 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:40:45.573 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:40:45.580 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:40:45.580 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:40:45.863 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:40:45.870 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:40:46.872 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:40:47.153 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: test_get_osds: wait_for_clean 2026-02-21T11:40:47.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:40:47.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:40:47.154 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:40:47.154 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:40:47.154 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:40:47.154 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:40:47.154 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:40:47.154 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:40:47.154 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:40:47.236 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:40:47.237 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:40:47.237 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:40:47.237 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:40:47.237 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:40:47.237 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:40:47.462 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids='0 2026-02-21T11:40:47.462 INFO:tasks.workunit.client.0.trial189.stderr:1' 2026-02-21T11:40:47.462 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:40:47.463 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:40:47.463 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:40:47.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836483 2026-02-21T11:40:47.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836483 2026-02-21T11:40:47.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T11:40:47.529 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:40:47.529 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-21T11:40:47.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=42949672963 2026-02-21T11:40:47.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 42949672963 2026-02-21T11:40:47.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836483 1-42949672963' 2026-02-21T11:40:47.592 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:40:47.593 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836483 2026-02-21T11:40:47.593 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:40:47.594 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:40:47.594 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836483 2026-02-21T11:40:47.594 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:40:47.595 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836483 2026-02-21T11:40:47.595 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T11:40:47.596 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836483 2026-02-21T11:40:47.596 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:40:47.822 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T11:40:47.822 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:40:48.823 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:40:48.823 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:40:49.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-21T11:40:49.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:40:49.053 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 1-42949672963 2026-02-21T11:40:49.054 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:40:49.054 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=1 2026-02-21T11:40:49.055 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 1-42949672963 2026-02-21T11:40:49.055 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:40:49.056 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=42949672963 2026-02-21T11:40:49.056 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.1 seq 42949672963' 2026-02-21T11:40:49.056 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.1 seq 42949672963 2026-02-21T11:40:49.056 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-21T11:40:49.288 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 42949672963 -lt 42949672963 2026-02-21T11:40:49.289 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:40:49.289 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:40:49.289 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:49.604 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T11:40:49.605 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:40:49.605 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:40:49.605 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:40:49.605 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:40:49.605 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:40:49.605 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:40:49.605 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:40:49.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T11:40:49.827 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:40:49.827 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:40:49.827 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:50.142 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T11:40:50.142 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:40:50.142 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:40:50.142 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: create_rbd_pool 2026-02-21T11:40:50.142 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:40:50.415 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:40:50.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:40:50.423 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:40:50.702 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:40:50.710 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:40:51.712 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:40:51.992 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: test_get_osds: get_osds rbd GROUP 2026-02-21T11:40:51.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2026-02-21T11:40:51.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=rbd 2026-02-21T11:40:51.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=GROUP 2026-02-21T11:40:51.993 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map rbd GROUP 2026-02-21T11:40:51.993 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-21T11:40:52.219 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-21T11:40:52.219 INFO:tasks.workunit.client.0.trial189.stderr:0' 2026-02-21T11:40:52.219 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-21T11:40:52.219 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: teardown td/ceph-helpers 2026-02-21T11:40:52.219 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:52.219 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:52.219 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:52.219 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:52.220 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:52.220 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:52.220 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:52.220 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:52.325 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:52.325 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:52.326 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:52.326 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:52.327 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:52.327 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:52.327 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:52.328 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:52.328 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:52.328 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:52.329 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:52.329 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:52.330 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:52.330 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:52.335 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:52.335 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:52.335 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:52.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:52.337 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:52.337 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:52.337 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:40:52.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_get_pg td/ceph-helpers 2026-02-21T11:40:52.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1091: test_get_pg: local dir=td/ceph-helpers 2026-02-21T11:40:52.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: test_get_pg: setup td/ceph-helpers 2026-02-21T11:40:52.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:40:52.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:40:52.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:52.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:52.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:52.338 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:52.338 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:52.338 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:52.339 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:52.339 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:52.339 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:52.339 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:52.340 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:52.340 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:52.341 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:52.341 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:52.341 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:52.342 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:52.342 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:52.342 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:52.342 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:52.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:52.344 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:52.344 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:52.345 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:52.345 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:52.345 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:52.345 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:52.346 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:52.346 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:52.346 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:40:52.347 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:40:52.347 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:52.347 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:52.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:40:52.349 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:40:52.349 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:40:52.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:40:52.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:40:52.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:52.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:40:52.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:40:52.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:40:52.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:40:52.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:40:52.350 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:52.369 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:40:52.369 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:52.369 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:52.369 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:52.369 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:52.369 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:52.370 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:52.370 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:52.393 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:40:52.395 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:40:52.395 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:52.395 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:52.395 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:40:52.395 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:40:52.396 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:52.396 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:52.396 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:52.396 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:52.396 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:52.396 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:52.396 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:52.396 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:52.396 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:40:52.437 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:40:52.437 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:52.437 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:52.437 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:40:52.437 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:40:52.437 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:52.437 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:52.437 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:52.438 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:52.438 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:52.438 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:52.439 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:52.439 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:52.439 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:40:52.480 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1095: test_get_pg: run_mgr td/ceph-helpers x 2026-02-21T11:40:52.480 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:40:52.480 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:40:52.480 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:40:52.480 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:40:52.480 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:40:52.480 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:40:52.578 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:40:52.578 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:52.578 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:52.579 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:52.579 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:52.579 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:52.579 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:52.579 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:52.580 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:52.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: test_get_pg: run_osd td/ceph-helpers 0 2026-02-21T11:40:52.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:40:52.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:40:52.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:40:52.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:40:52.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:40:52.597 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:40:52.598 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:40:52.598 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:40:52.598 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:40:52.598 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:40:52.598 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:40:52.598 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:40:52.598 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:40:52.598 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:40:52.598 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:40:52.598 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:52.598 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:52.598 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:52.599 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:52.599 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:52.599 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:52.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:52.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:40:52.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:40:52.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:40:52.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:40:52.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:40:52.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:40:52.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:40:52.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:40:52.599 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:40:52.600 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:40:52.600 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:40:52.600 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:40:52.600 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:40:52.600 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2918d059-c5dc-4d8c-89c1-5740dde80332 2026-02-21T11:40:52.600 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 2918d059-c5dc-4d8c-89c1-5740dde80332 2026-02-21T11:40:52.601 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2918d059-c5dc-4d8c-89c1-5740dde80332' 2026-02-21T11:40:52.601 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:40:52.607 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDEmZlpolIgJBAAM5QWi0lUsrneqVQLjCDF4Q== 2026-02-21T11:40:52.607 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDEmZlpolIgJBAAM5QWi0lUsrneqVQLjCDF4Q=="}' 2026-02-21T11:40:52.607 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2918d059-c5dc-4d8c-89c1-5740dde80332 -i td/ceph-helpers/0/new.json 2026-02-21T11:40:52.709 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:52.715 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:40:52.716 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDEmZlpolIgJBAAM5QWi0lUsrneqVQLjCDF4Q== --osd-uuid 2918d059-c5dc-4d8c-89c1-5740dde80332 2026-02-21T11:40:52.731 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:52.729+0000 7f5a07e2d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:52.733 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:52.731+0000 7f5a07e2d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:52.734 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:52.732+0000 7f5a07e2d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:52.734 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:52.732+0000 7f5a07e2d780 -1 bdev(0x36dac00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:40:52.734 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:52.732+0000 7f5a07e2d780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:40:53.136 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:40:53.136 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:40:53.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:40:53.137 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:40:53.137 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:40:53.239 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:40:53.240 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:40:53.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:40:53.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:40:53.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:40:53.240 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:40:53.254 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:53.252+0000 7f1b94238780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:53.255 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:53.254+0000 7f1b94238780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:53.256 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:53.255+0000 7f1b94238780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:53.339 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:40:53.340 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:40:53.340 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:40:53.340 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:40:53.340 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:40:53.340 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:53.340 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:53.341 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:40:53.341 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:53.341 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:53.436 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:40:53.441 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:53.440+0000 7f1b94238780 -1 Falling back to public interface 2026-02-21T11:40:53.791 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:53.789+0000 7f1b94238780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:40:54.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:40:54.437 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:54.438 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:40:54.438 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:40:54.439 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:54.439 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:54.536 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2009038619,v1:127.0.0.1:6801/2009038619] [v2:127.0.0.1:6802/2009038619,v1:127.0.0.1:6803/2009038619] exists,up 2918d059-c5dc-4d8c-89c1-5740dde80332 2026-02-21T11:40:54.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:40:54.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:40:54.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:40:54.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1097: test_get_pg: create_rbd_pool 2026-02-21T11:40:54.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:40:54.630 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:40:54.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:40:54.636 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:40:54.813 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:40:54.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:40:55.820 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:40:56.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: test_get_pg: wait_for_clean 2026-02-21T11:40:56.102 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:40:56.102 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:40:56.102 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:40:56.102 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:40:56.102 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:40:56.102 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:40:56.102 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:40:56.102 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:40:56.102 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:40:56.185 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:40:56.185 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:40:56.185 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:40:56.185 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:40:56.185 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:40:56.185 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:40:56.411 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:40:56.411 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:40:56.411 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:40:56.412 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:40:56.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836483 2026-02-21T11:40:56.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836483 2026-02-21T11:40:56.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T11:40:56.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:40:56.478 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836483 2026-02-21T11:40:56.478 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:40:56.479 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:40:56.479 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836483 2026-02-21T11:40:56.480 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:40:56.481 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836483 2026-02-21T11:40:56.481 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T11:40:56.481 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836483 2026-02-21T11:40:56.481 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:40:56.710 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T11:40:56.710 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:40:56.710 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:40:56.711 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:57.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T11:40:57.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:40:57.025 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:40:57.025 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:40:57.025 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:40:57.025 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:40:57.025 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:40:57.025 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:40:57.246 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T11:40:57.246 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:40:57.247 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:40:57.247 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:40:57.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T11:40:57.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:40:57.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:40:57.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: test_get_pg: get_pg rbd GROUP 2026-02-21T11:40:57.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-21T11:40:57.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=GROUP 2026-02-21T11:40:57.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2026-02-21T11:40:57.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd GROUP 2026-02-21T11:40:57.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-21T11:40:57.800 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: teardown td/ceph-helpers 2026-02-21T11:40:57.800 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:57.800 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:57.800 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:57.800 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:57.800 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:57.800 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:57.800 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:57.800 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:57.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:57.907 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:57.907 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:57.908 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:57.909 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:57.909 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:57.909 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:57.910 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:57.910 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:57.910 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:57.910 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:57.911 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:57.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:57.912 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:57.915 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:57.915 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:57.915 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:57.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:57.917 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:57.917 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:57.917 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:40:57.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_get_primary td/ceph-helpers 2026-02-21T11:40:57.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: test_get_primary: local dir=td/ceph-helpers 2026-02-21T11:40:57.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: test_get_primary: setup td/ceph-helpers 2026-02-21T11:40:57.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:40:57.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:40:57.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:40:57.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:40:57.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:40:57.918 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:40:57.918 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:40:57.918 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:40:57.919 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:40:57.919 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:40:57.919 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:40:57.920 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:40:57.920 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:40:57.921 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:40:57.921 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:40:57.921 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:40:57.921 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:40:57.922 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:57.922 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:40:57.922 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:40:57.922 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:40:57.923 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:40:57.924 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:40:57.924 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:40:57.925 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:40:57.925 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:57.925 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:57.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:40:57.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:40:57.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:40:57.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:40:57.927 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:40:57.927 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:57.927 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:57.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:40:57.928 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:40:57.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:40:57.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:40:57.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:40:57.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:57.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:40:57.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:40:57.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:40:57.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:40:57.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:40:57.929 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:57.947 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:40:57.948 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:57.948 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:57.948 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:57.948 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:57.948 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:57.948 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:57.948 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:40:57.973 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:40:57.973 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:40:57.973 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:57.973 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:57.974 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:40:57.974 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:40:57.974 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:57.974 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:57.974 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:57.975 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:57.975 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:57.975 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:57.975 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:57.975 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:57.975 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:40:58.017 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:40:58.017 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:40:58.017 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:40:58.017 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:40:58.017 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:40:58.017 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:40:58.017 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:40:58.017 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:40:58.018 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:40:58.018 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:58.018 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:58.018 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:40:58.018 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:40:58.018 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:40:58.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: test_get_primary: local osd=0 2026-02-21T11:40:58.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: test_get_primary: run_mgr td/ceph-helpers x 2026-02-21T11:40:58.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:40:58.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:40:58.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:40:58.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:40:58.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:40:58.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:40:58.159 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:40:58.159 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:58.159 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:58.160 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:58.160 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:58.160 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:58.160 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:58.160 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:58.160 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:40:58.178 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: test_get_primary: run_osd td/ceph-helpers 0 2026-02-21T11:40:58.178 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:40:58.178 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:40:58.178 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:40:58.178 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:40:58.178 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:40:58.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:40:58.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:40:58.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:40:58.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:40:58.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:40:58.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:40:58.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:40:58.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:40:58.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:40:58.179 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:40:58.179 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:40:58.179 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:40:58.180 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:40:58.180 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:40:58.180 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:40:58.180 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:58.180 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:40:58.180 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:40:58.180 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:40:58.180 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:40:58.180 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:40:58.180 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:40:58.180 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:40:58.180 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:40:58.180 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:40:58.181 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:40:58.181 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:40:58.181 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:40:58.181 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:40:58.181 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:40:58.181 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dfdb1665-a644-40a1-855e-f0faa7197136 2026-02-21T11:40:58.181 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 dfdb1665-a644-40a1-855e-f0faa7197136' 2026-02-21T11:40:58.181 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 dfdb1665-a644-40a1-855e-f0faa7197136 2026-02-21T11:40:58.182 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:40:58.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDKmZlpRa0XCxAAj2e8bhZARQE3gKiVvYRAbA== 2026-02-21T11:40:58.187 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDKmZlpRa0XCxAAj2e8bhZARQE3gKiVvYRAbA=="}' 2026-02-21T11:40:58.188 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dfdb1665-a644-40a1-855e-f0faa7197136 -i td/ceph-helpers/0/new.json 2026-02-21T11:40:58.288 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:58.294 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:40:58.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKmZlpRa0XCxAAj2e8bhZARQE3gKiVvYRAbA== --osd-uuid dfdb1665-a644-40a1-855e-f0faa7197136 2026-02-21T11:40:58.311 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:58.309+0000 7fc13cc0e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:58.312 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:58.310+0000 7fc13cc0e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:58.313 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:58.311+0000 7fc13cc0e780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:58.314 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:58.311+0000 7fc13cc0e780 -1 bdev(0x2bdec00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:40:58.314 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:58.312+0000 7fc13cc0e780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:40:58.749 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:40:58.749 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:40:58.750 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:40:58.751 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:40:58.751 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:40:58.852 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:40:58.853 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:40:58.853 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:40:58.854 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:40:58.854 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:40:58.854 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:40:58.867 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:58.865+0000 7f3aff1d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:58.868 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:58.867+0000 7f3aff1d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:58.869 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:58.867+0000 7f3aff1d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:40:58.948 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:40:58.949 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:40:58.949 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:40:58.949 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:40:58.949 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:40:58.949 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:40:58.950 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:40:58.951 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:40:58.951 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:40:58.951 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:40:59.047 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:40:59.059 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:59.058+0000 7f3aff1d8780 -1 Falling back to public interface 2026-02-21T11:40:59.393 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:40:59.391+0000 7f3aff1d8780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:41:00.048 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:41:00.048 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:41:00.049 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:41:00.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:41:00.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:41:00.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:41:00.145 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3303918482,v1:127.0.0.1:6801/3303918482] [v2:127.0.0.1:6802/3303918482,v1:127.0.0.1:6803/3303918482] exists,up dfdb1665-a644-40a1-855e-f0faa7197136 2026-02-21T11:41:00.146 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:41:00.147 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:41:00.147 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:41:00.147 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: test_get_primary: create_rbd_pool 2026-02-21T11:41:00.147 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:41:00.241 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:41:00.246 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:41:00.246 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:41:00.559 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:41:00.567 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:41:01.568 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:41:01.850 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: test_get_primary: wait_for_clean 2026-02-21T11:41:01.850 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:41:01.850 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:41:01.850 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:41:01.850 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:41:01.850 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:01.850 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:01.850 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:01.851 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:01.851 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:01.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:41:01.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:41:01.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:41:01.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:41:01.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:41:01.933 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:41:02.159 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:41:02.159 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:41:02.159 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:41:02.160 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:41:02.223 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836483 2026-02-21T11:41:02.223 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836483 2026-02-21T11:41:02.223 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T11:41:02.224 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:41:02.224 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836483 2026-02-21T11:41:02.224 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:41:02.224 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:41:02.225 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836483 2026-02-21T11:41:02.225 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:41:02.226 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836483 2026-02-21T11:41:02.226 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T11:41:02.226 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836483 2026-02-21T11:41:02.227 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:02.451 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T11:41:02.451 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:41:03.452 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:41:03.453 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:03.679 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T11:41:03.679 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:41:04.680 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T11:41:04.680 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:04.905 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T11:41:04.905 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:41:04.906 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:41:04.906 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:41:05.217 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T11:41:05.218 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:41:05.218 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:41:05.218 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:41:05.218 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:41:05.218 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:41:05.218 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:41:05.218 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:41:05.441 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T11:41:05.442 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:41:05.442 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:41:05.442 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:41:05.756 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T11:41:05.756 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:41:05.757 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:41:05.757 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: get_primary rbd GROUP 2026-02-21T11:41:05.757 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-21T11:41:05.757 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2026-02-21T11:41:05.757 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2026-02-21T11:41:05.757 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-21T11:41:05.985 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: test 0 = 0 2026-02-21T11:41:05.985 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: test_get_primary: teardown td/ceph-helpers 2026-02-21T11:41:05.985 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:41:05.985 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:41:05.985 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:41:05.986 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:05.986 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:05.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:05.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:05.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:06.092 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:41:06.093 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:41:06.093 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:41:06.093 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:41:06.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:41:06.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:41:06.095 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:41:06.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:06.096 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:41:06.097 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:41:06.097 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:06.097 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:41:06.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:41:06.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:41:06.102 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:41:06.102 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:06.102 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:06.102 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:41:06.103 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:41:06.103 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:41:06.103 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:41:06.103 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_get_timeout_delays td/ceph-helpers 2026-02-21T11:41:06.103 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: get_timeout_delays 1 2026-02-21T11:41:06.104 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:06.104 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:06.104 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:06.105 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:06.105 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:06.114 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: test '1 ' = '1 ' 2026-02-21T11:41:06.114 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: get_timeout_delays 5 2026-02-21T11:41:06.115 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:06.115 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:06.115 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:06.115 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:06.116 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:06.132 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2026-02-21T11:41:06.132 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: get_timeout_delays 6 2026-02-21T11:41:06.133 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:06.133 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:06.133 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:06.134 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:06.134 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:06.151 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2026-02-21T11:41:06.152 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: get_timeout_delays 7 2026-02-21T11:41:06.152 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:06.152 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:06.152 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:06.152 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:06.152 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:06.174 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2026-02-21T11:41:06.175 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: get_timeout_delays 8 2026-02-21T11:41:06.175 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:06.176 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:06.176 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:06.176 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:06.176 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:06.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2026-02-21T11:41:06.200 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: get_timeout_delays 1 .1 2026-02-21T11:41:06.201 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:06.201 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:06.201 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:06.201 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:06.201 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:06.225 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2026-02-21T11:41:06.226 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: get_timeout_delays 1.5 .1 2026-02-21T11:41:06.227 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:06.227 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:06.227 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:06.227 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:06.227 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:06.255 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2026-02-21T11:41:06.256 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: get_timeout_delays 5 .1 2026-02-21T11:41:06.256 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:06.256 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:06.256 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:06.256 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:06.256 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:06.292 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2026-02-21T11:41:06.293 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1629: test_get_timeout_delays: get_timeout_delays 6 .1 2026-02-21T11:41:06.293 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:06.293 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:06.293 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:06.293 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:06.294 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:06.332 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1629: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2026-02-21T11:41:06.332 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1630: test_get_timeout_delays: get_timeout_delays 6.3 .1 2026-02-21T11:41:06.333 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:06.333 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:06.333 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:06.333 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:06.333 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:06.368 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1630: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2026-02-21T11:41:06.368 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: test_get_timeout_delays: get_timeout_delays 20 .1 2026-02-21T11:41:06.369 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:06.369 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:06.369 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:06.370 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:06.370 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:06.418 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2026-02-21T11:41:06.420 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1632: test_get_timeout_delays: get_timeout_delays 300 .1 0 2026-02-21T11:41:06.420 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:06.420 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:06.420 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:06.420 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:06.420 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:06.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1632: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2026-02-21T11:41:06.493 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: test_get_timeout_delays: get_timeout_delays 300 .1 10 2026-02-21T11:41:06.494 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:06.494 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:06.494 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:06.494 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:06.494 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:06.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2026-02-21T11:41:06.725 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:41:06.725 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_is_clean td/ceph-helpers 2026-02-21T11:41:06.725 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: test_is_clean: local dir=td/ceph-helpers 2026-02-21T11:41:06.725 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: test_is_clean: setup td/ceph-helpers 2026-02-21T11:41:06.725 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:41:06.725 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:41:06.725 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:41:06.725 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:41:06.725 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:41:06.725 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:06.725 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:06.726 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:06.726 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:06.726 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:06.726 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:41:06.727 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:41:06.727 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:41:06.727 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:41:06.727 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:41:06.727 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:41:06.728 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:41:06.728 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:06.729 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:41:06.729 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:41:06.729 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:06.730 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:41:06.730 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:41:06.731 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:41:06.732 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:41:06.732 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:06.732 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:06.732 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:41:06.733 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:41:06.733 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:41:06.733 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:41:06.734 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:41:06.734 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:06.734 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:06.734 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:41:06.735 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:41:06.736 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:41:06.736 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:41:06.736 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:41:06.736 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:41:06.737 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:41:06.737 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:41:06.737 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:41:06.737 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:41:06.737 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:41:06.737 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:41:06.755 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:41:06.755 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:06.755 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:06.755 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:06.755 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:06.755 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:06.756 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:06.756 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:41:06.779 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:41:06.781 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:41:06.781 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:41:06.781 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:41:06.781 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:41:06.781 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:41:06.782 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:41:06.782 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:41:06.782 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:41:06.782 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:41:06.782 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:06.782 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:06.782 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:41:06.782 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:41:06.782 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:41:06.825 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:41:06.825 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:41:06.825 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:41:06.825 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:41:06.826 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:41:06.826 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:41:06.826 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:41:06.826 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:41:06.826 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:41:06.826 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:06.826 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:06.826 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:41:06.826 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:41:06.826 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:41:06.866 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: test_is_clean: run_mgr td/ceph-helpers x 2026-02-21T11:41:06.866 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:41:06.866 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:41:06.866 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:41:06.867 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:41:06.867 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:41:06.867 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:41:06.966 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:41:06.966 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:06.966 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:06.966 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:06.966 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:06.967 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:06.967 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:06.967 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:41:06.967 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:41:06.985 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: test_is_clean: run_osd td/ceph-helpers 0 2026-02-21T11:41:06.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:41:06.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:41:06.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:41:06.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:41:06.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:41:06.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:41:06.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:41:06.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:41:06.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:41:06.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:41:06.986 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:41:06.987 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:41:06.987 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:41:06.987 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:41:06.987 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:41:06.987 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:06.987 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:06.987 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:06.987 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:06.987 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:06.987 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:06.987 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:06.987 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:41:06.987 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:41:06.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:41:06.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:41:06.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:41:06.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:41:06.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:41:06.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:41:06.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:41:06.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:41:06.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:41:06.988 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:41:06.988 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:41:06.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8fc539d7-86c5-4396-9310-5f57c64a2628 2026-02-21T11:41:06.989 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 8fc539d7-86c5-4396-9310-5f57c64a2628 2026-02-21T11:41:06.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8fc539d7-86c5-4396-9310-5f57c64a2628' 2026-02-21T11:41:06.990 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:41:06.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDSmZlpZ5FQOxAAhm82B8lq4DX+H4Q4BmbrzA== 2026-02-21T11:41:06.997 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDSmZlpZ5FQOxAAhm82B8lq4DX+H4Q4BmbrzA=="}' 2026-02-21T11:41:06.997 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8fc539d7-86c5-4396-9310-5f57c64a2628 -i td/ceph-helpers/0/new.json 2026-02-21T11:41:07.098 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:41:07.105 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:41:07.106 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSmZlpZ5FQOxAAhm82B8lq4DX+H4Q4BmbrzA== --osd-uuid 8fc539d7-86c5-4396-9310-5f57c64a2628 2026-02-21T11:41:07.123 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:07.121+0000 7fc3b87d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:07.124 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:07.122+0000 7fc3b87d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:07.125 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:07.123+0000 7fc3b87d8780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:07.125 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:07.123+0000 7fc3b87d8780 -1 bdev(0x364cc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:41:07.126 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:07.123+0000 7fc3b87d8780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:41:07.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:41:07.528 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:41:07.529 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:41:07.529 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:41:07.530 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:41:07.630 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:41:07.631 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:41:07.632 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:41:07.632 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:41:07.632 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:41:07.632 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:41:07.645 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:07.643+0000 7f8d94a0d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:07.647 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:07.645+0000 7f8d94a0d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:07.648 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:07.646+0000 7f8d94a0d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:07.728 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:41:07.728 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:41:07.728 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:41:07.728 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:41:07.728 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:41:07.728 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:41:07.728 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:41:07.729 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:41:07.729 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:41:07.729 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:41:07.825 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:41:07.857 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:07.856+0000 7f8d94a0d780 -1 Falling back to public interface 2026-02-21T11:41:08.243 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:08.242+0000 7f8d94a0d780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:41:08.826 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:41:08.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:41:08.827 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:41:08.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:41:08.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:41:08.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:41:08.926 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2045174707,v1:127.0.0.1:6801/2045174707] [v2:127.0.0.1:6802/2045174707,v1:127.0.0.1:6803/2045174707] exists,up 8fc539d7-86c5-4396-9310-5f57c64a2628 2026-02-21T11:41:08.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:41:08.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:41:08.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:41:08.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1558: test_is_clean: create_rbd_pool 2026-02-21T11:41:08.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:41:09.022 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:41:09.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:41:09.029 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:41:09.336 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:41:09.344 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:41:10.345 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:41:10.624 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: test_is_clean: wait_for_clean 2026-02-21T11:41:10.624 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:41:10.624 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:41:10.625 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:41:10.625 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:41:10.625 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:10.625 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:10.625 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:10.625 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:10.626 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:10.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:41:10.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:41:10.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:41:10.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:41:10.709 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:41:10.709 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:41:10.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:41:10.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:41:10.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:41:10.931 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:41:10.995 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836483 2026-02-21T11:41:10.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836483 2026-02-21T11:41:10.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T11:41:10.996 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:41:10.996 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836483 2026-02-21T11:41:10.996 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:41:10.997 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:41:10.997 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836483 2026-02-21T11:41:10.997 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:41:10.998 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836483 2026-02-21T11:41:10.998 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T11:41:10.998 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836483 2026-02-21T11:41:10.999 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:11.230 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T11:41:11.230 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:41:12.231 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:41:12.231 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:12.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T11:41:12.455 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:41:13.456 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T11:41:13.457 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:13.681 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T11:41:13.682 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:41:13.682 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:41:13.682 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:41:14.009 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T11:41:14.009 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:41:14.009 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:41:14.009 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:41:14.009 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:41:14.009 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:41:14.009 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:41:14.009 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:41:14.230 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T11:41:14.230 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:41:14.230 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:41:14.231 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:41:14.546 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T11:41:14.546 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:41:14.546 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:41:14.546 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1560: test_is_clean: is_clean 2026-02-21T11:41:14.547 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1546: is_clean: get_num_pgs 2026-02-21T11:41:14.547 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:41:14.547 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:41:14.858 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1546: is_clean: num_pgs=4 2026-02-21T11:41:14.858 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1547: is_clean: test 4 '!=' 0 2026-02-21T11:41:14.858 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: is_clean: get_num_active_clean 2026-02-21T11:41:14.858 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:41:14.858 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:41:14.858 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:41:14.858 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:41:14.858 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:41:15.082 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: is_clean: test 4 = 4 2026-02-21T11:41:15.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: teardown td/ceph-helpers 2026-02-21T11:41:15.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:41:15.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:41:15.083 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:41:15.084 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:15.084 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:15.084 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:15.084 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:15.084 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:15.188 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:41:15.188 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:41:15.189 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:41:15.190 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:41:15.191 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:41:15.191 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:41:15.192 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:41:15.193 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:15.193 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:41:15.193 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:41:15.194 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:15.194 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:41:15.195 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:41:15.195 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:41:15.198 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:41:15.198 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:15.199 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:15.199 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:41:15.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:41:15.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:41:15.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:41:15.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_kill_daemon td/ceph-helpers 2026-02-21T11:41:15.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: test_kill_daemon: local dir=td/ceph-helpers 2026-02-21T11:41:15.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:267: test_kill_daemon: setup td/ceph-helpers 2026-02-21T11:41:15.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:41:15.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:41:15.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:41:15.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:41:15.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:41:15.202 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:15.202 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:15.202 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:15.202 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:15.202 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:15.203 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:41:15.203 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:41:15.203 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:41:15.203 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:41:15.204 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:41:15.204 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:41:15.204 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:41:15.205 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:15.205 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:41:15.205 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:41:15.205 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:15.206 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:41:15.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:41:15.207 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:41:15.208 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:41:15.208 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:15.208 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:15.208 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:41:15.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:41:15.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:41:15.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:41:15.210 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:41:15.210 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:15.210 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:15.210 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:41:15.211 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:41:15.213 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:41:15.213 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:41:15.213 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:41:15.213 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:41:15.213 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:41:15.213 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:41:15.213 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:41:15.213 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:41:15.213 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:41:15.213 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:41:15.230 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:41:15.230 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:15.230 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:15.230 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:15.230 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:15.230 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:15.230 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:15.230 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:41:15.254 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:41:15.255 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:41:15.255 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:41:15.255 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:41:15.255 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:41:15.255 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:41:15.255 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:41:15.255 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:41:15.256 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:41:15.256 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:41:15.256 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:15.256 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:15.256 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:41:15.256 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:41:15.256 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:41:15.299 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:41:15.299 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:41:15.299 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:41:15.299 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:41:15.299 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:41:15.299 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:41:15.299 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:41:15.299 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:41:15.300 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:41:15.300 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:15.300 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:15.300 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:41:15.300 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:41:15.300 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:41:15.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: test_kill_daemon: run_mgr td/ceph-helpers x 2026-02-21T11:41:15.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:41:15.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:41:15.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:41:15.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:41:15.343 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:41:15.344 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:41:15.440 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:41:15.440 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:15.441 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:15.441 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:15.441 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:15.441 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:15.441 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:15.441 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:41:15.441 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:41:15.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:270: test_kill_daemon: run_osd td/ceph-helpers 0 2026-02-21T11:41:15.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:41:15.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:41:15.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:41:15.459 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:41:15.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:41:15.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:41:15.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:41:15.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:41:15.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:41:15.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:41:15.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:41:15.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:41:15.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:41:15.460 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:41:15.460 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:41:15.460 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:15.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:15.461 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:15.461 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:15.461 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:15.461 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:15.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:15.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:41:15.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:41:15.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:41:15.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:41:15.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:41:15.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:41:15.461 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:41:15.462 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:41:15.462 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:41:15.462 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:41:15.462 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:41:15.462 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:41:15.462 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:41:15.463 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7b8b9238-bf95-4259-b8f2-331d88fca5c7 2026-02-21T11:41:15.463 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7b8b9238-bf95-4259-b8f2-331d88fca5c7' 2026-02-21T11:41:15.463 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 7b8b9238-bf95-4259-b8f2-331d88fca5c7 2026-02-21T11:41:15.463 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:41:15.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDbmZlp3oX4GxAAh/amCmJ3zf2JsqR4wbdnJA== 2026-02-21T11:41:15.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDbmZlp3oX4GxAAh/amCmJ3zf2JsqR4wbdnJA=="}' 2026-02-21T11:41:15.470 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7b8b9238-bf95-4259-b8f2-331d88fca5c7 -i td/ceph-helpers/0/new.json 2026-02-21T11:41:15.572 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:41:15.579 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:41:15.580 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbmZlp3oX4GxAAh/amCmJ3zf2JsqR4wbdnJA== --osd-uuid 7b8b9238-bf95-4259-b8f2-331d88fca5c7 2026-02-21T11:41:15.595 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:15.593+0000 7f959d847780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:15.596 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:15.595+0000 7f959d847780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:15.597 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:15.596+0000 7f959d847780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:15.598 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:15.596+0000 7f959d847780 -1 bdev(0x34c8c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:41:15.598 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:15.596+0000 7f959d847780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:41:15.992 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:41:15.992 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:41:15.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:41:15.993 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:41:15.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:41:16.098 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:41:16.099 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:41:16.100 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:41:16.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:41:16.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:41:16.101 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:41:16.112 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:16.110+0000 7f361de0c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:16.113 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:16.111+0000 7f361de0c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:16.114 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:16.112+0000 7f361de0c780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:16.196 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:41:16.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:41:16.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:41:16.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:41:16.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:41:16.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:41:16.197 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:41:16.198 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:41:16.198 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:41:16.198 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:41:16.294 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:41:16.299 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:16.298+0000 7f361de0c780 -1 Falling back to public interface 2026-02-21T11:41:16.705 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:16.703+0000 7f361de0c780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:41:17.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:41:17.295 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:41:17.296 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:41:17.296 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:41:17.297 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:41:17.297 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:41:17.395 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/617174968,v1:127.0.0.1:6801/617174968] [v2:127.0.0.1:6802/617174968,v1:127.0.0.1:6803/617174968] exists,up 7b8b9238-bf95-4259-b8f2-331d88fca5c7 2026-02-21T11:41:17.396 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:41:17.396 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:41:17.396 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:41:17.396 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:272: test_kill_daemon: name_prefix=osd 2026-02-21T11:41:17.397 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: find td/ceph-helpers 2026-02-21T11:41:17.397 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: grep 'osd[^/]*\.pid' 2026-02-21T11:41:17.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2026-02-21T11:41:17.397 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:280: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2026-02-21T11:41:17.398 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/osd.0.pid 2026-02-21T11:41:17.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=159004 2026-02-21T11:41:17.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=0 2026-02-21T11:41:17.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local delays=1 2026-02-21T11:41:17.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-21T11:41:17.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T11:41:17.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 159004 2026-02-21T11:41:17.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-21T11:41:17.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-21T11:41:17.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 1 2026-02-21T11:41:18.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T11:41:18.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 159004 2026-02-21T11:41:18.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-21T11:41:18.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-21T11:41:18.399 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0 2026-02-21T11:41:18.400 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 1 2026-02-21T11:41:18.401 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:284: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2026-02-21T11:41:18.401 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/osd.0.pid 2026-02-21T11:41:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=159004 2026-02-21T11:41:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2026-02-21T11:41:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2026-02-21T11:41:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-21T11:41:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T11:41:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 159004 2026-02-21T11:41:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-21T11:41:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-21T11:41:18.402 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2026-02-21T11:41:18.503 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T11:41:18.504 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 159004 2026-02-21T11:41:18.504 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2026-02-21T11:41:18.504 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2026-02-21T11:41:18.504 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2026-02-21T11:41:18.504 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:287: test_kill_daemon: name_prefix=mgr 2026-02-21T11:41:18.504 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: find td/ceph-helpers 2026-02-21T11:41:18.504 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: grep 'mgr[^/]*\.pid' 2026-02-21T11:41:18.505 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2026-02-21T11:41:18.505 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:292: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2026-02-21T11:41:18.505 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2026-02-21T11:41:18.506 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=158149 2026-02-21T11:41:18.506 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2026-02-21T11:41:18.506 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2026-02-21T11:41:18.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-21T11:41:18.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T11:41:18.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 158149 2026-02-21T11:41:18.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-21T11:41:18.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-21T11:41:18.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2026-02-21T11:41:18.608 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T11:41:18.608 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 158149 2026-02-21T11:41:18.608 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2026-02-21T11:41:18.608 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2026-02-21T11:41:18.608 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2026-02-21T11:41:18.608 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:295: test_kill_daemon: name_prefix=mon 2026-02-21T11:41:18.608 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: find td/ceph-helpers 2026-02-21T11:41:18.608 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: grep 'mon[^/]*\.pid' 2026-02-21T11:41:18.610 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2026-02-21T11:41:18.610 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:300: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2026-02-21T11:41:18.611 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/mon.a.pid 2026-02-21T11:41:18.612 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=158064 2026-02-21T11:41:18.612 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2026-02-21T11:41:18.612 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2026-02-21T11:41:18.612 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-21T11:41:18.612 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T11:41:18.612 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 158064 2026-02-21T11:41:18.612 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-21T11:41:18.612 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-21T11:41:18.612 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2026-02-21T11:41:18.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T11:41:18.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 158064 2026-02-21T11:41:18.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-21T11:41:18.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-21T11:41:18.714 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.2 2026-02-21T11:41:18.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-21T11:41:18.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 158064 2026-02-21T11:41:18.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2026-02-21T11:41:18.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2026-02-21T11:41:18.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2026-02-21T11:41:18.916 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: timeout 5 ceph status 2026-02-21T11:41:23.919 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: teardown td/ceph-helpers 2026-02-21T11:41:23.919 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:41:23.919 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:41:23.920 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:41:23.920 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:23.920 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:23.920 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:23.920 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:23.920 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:23.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:41:23.925 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:41:23.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:41:23.926 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:41:23.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:41:23.927 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:41:23.927 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:41:23.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:23.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:41:23.928 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:41:23.928 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:23.929 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:41:23.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:41:23.930 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:41:23.933 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:41:23.933 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:23.933 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:23.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:41:23.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:41:23.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:41:23.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:41:23.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_kill_daemons td/ceph-helpers 2026-02-21T11:41:23.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:357: test_kill_daemons: local dir=td/ceph-helpers 2026-02-21T11:41:23.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:358: test_kill_daemons: setup td/ceph-helpers 2026-02-21T11:41:23.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:41:23.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:41:23.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:41:23.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:41:23.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:41:23.935 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:23.935 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:23.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:23.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:23.936 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:23.937 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:41:23.937 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:41:23.938 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:41:23.938 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:41:23.939 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:41:23.939 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:41:23.939 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:41:23.940 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:23.940 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:41:23.940 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:41:23.940 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:23.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:41:23.942 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:41:23.942 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:41:23.943 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:41:23.943 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:23.943 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:23.943 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:41:23.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:41:23.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:41:23.944 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:41:23.945 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:41:23.945 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:23.945 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:23.945 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:41:23.946 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:41:23.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:41:23.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:41:23.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:41:23.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:359: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:41:23.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:41:23.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:41:23.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:41:23.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:41:23.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:41:23.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:41:23.965 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:41:23.965 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:23.965 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:23.965 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:23.965 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:23.965 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:23.965 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:23.965 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:41:23.991 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:41:23.991 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:41:23.991 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:41:23.991 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:41:23.991 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:41:23.991 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:41:23.992 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:41:23.992 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:41:23.992 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:41:23.992 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:41:23.992 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:23.992 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:23.992 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:41:23.992 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:41:23.992 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:41:24.037 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:41:24.037 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:41:24.037 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:41:24.038 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:41:24.038 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:41:24.038 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:41:24.038 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:41:24.038 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:41:24.038 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:41:24.038 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:24.039 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:24.039 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:41:24.039 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:41:24.039 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:41:24.079 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:360: test_kill_daemons: run_mgr td/ceph-helpers x 2026-02-21T11:41:24.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:41:24.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:41:24.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:41:24.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:41:24.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:41:24.080 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:41:24.180 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:41:24.180 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:24.180 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:24.180 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:24.180 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:24.181 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:24.181 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:24.181 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:41:24.181 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:361: test_kill_daemons: run_osd td/ceph-helpers 0 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:41:24.200 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:41:24.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:41:24.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:41:24.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:41:24.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:41:24.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:41:24.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:41:24.201 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:41:24.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=77198d5a-5259-438b-9250-b8554e4d17cc 2026-02-21T11:41:24.201 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 77198d5a-5259-438b-9250-b8554e4d17cc' 2026-02-21T11:41:24.201 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 77198d5a-5259-438b-9250-b8554e4d17cc 2026-02-21T11:41:24.201 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:41:24.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDkmZlpzTBsDBAAPoSdkKVufIdmHGNFII73Hg== 2026-02-21T11:41:24.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDkmZlpzTBsDBAAPoSdkKVufIdmHGNFII73Hg=="}' 2026-02-21T11:41:24.209 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 77198d5a-5259-438b-9250-b8554e4d17cc -i td/ceph-helpers/0/new.json 2026-02-21T11:41:24.312 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:41:24.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:41:24.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkmZlpzTBsDBAAPoSdkKVufIdmHGNFII73Hg== --osd-uuid 77198d5a-5259-438b-9250-b8554e4d17cc 2026-02-21T11:41:24.335 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:24.333+0000 7f4ff3a15780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:24.336 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:24.335+0000 7f4ff3a15780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:24.337 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:24.335+0000 7f4ff3a15780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:24.338 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:24.336+0000 7f4ff3a15780 -1 bdev(0x3826c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:41:24.338 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:24.336+0000 7f4ff3a15780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:41:24.770 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:41:24.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:41:24.771 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:41:24.772 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:41:24.772 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:41:24.876 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:41:24.877 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:41:24.877 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:41:24.878 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:41:24.878 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:41:24.878 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:41:24.890 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:24.889+0000 7fb63ba13780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:24.892 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:24.890+0000 7fb63ba13780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:24.893 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:24.891+0000 7fb63ba13780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:24.975 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:41:24.976 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:41:24.976 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:41:24.976 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:41:24.976 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:41:24.976 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:41:24.976 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:41:24.977 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:41:24.977 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:41:24.977 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:41:25.074 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:41:25.099 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:25.097+0000 7fb63ba13780 -1 Falling back to public interface 2026-02-21T11:41:25.464 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:25.462+0000 7fb63ba13780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:41:26.075 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:41:26.075 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:41:26.075 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:41:26.076 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:41:26.076 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:41:26.076 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:41:26.172 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/389728368,v1:127.0.0.1:6801/389728368] [v2:127.0.0.1:6802/389728368,v1:127.0.0.1:6803/389728368] exists,up 77198d5a-5259-438b-9250-b8554e4d17cc 2026-02-21T11:41:26.173 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:41:26.173 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:41:26.174 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:41:26.174 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2026-02-21T11:41:26.174 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:26.174 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:26.174 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:26.174 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:26.174 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:27.180 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 1 2026-02-21T11:41:27.180 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:372: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2026-02-21T11:41:27.181 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:27.181 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:27.181 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:27.181 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:27.181 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:27.287 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:41:27.287 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:376: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2026-02-21T11:41:27.287 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:27.287 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:27.287 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:27.287 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:27.287 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:27.393 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:41:27.393 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:380: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2026-02-21T11:41:27.393 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:27.393 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:27.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:27.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:27.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:27.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:41:27.500 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: timeout 5 ceph status 2026-02-21T11:41:32.505 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:382: test_kill_daemons: teardown td/ceph-helpers 2026-02-21T11:41:32.506 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:41:32.506 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:41:32.506 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:41:32.506 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:32.506 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:32.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:32.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:32.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:32.509 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:41:32.509 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:41:32.510 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:41:32.511 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:41:32.513 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:41:32.513 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:41:32.513 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:41:32.513 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:32.513 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:41:32.514 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:41:32.514 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:32.515 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:41:32.515 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:41:32.515 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:41:32.518 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:41:32.518 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:32.518 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:32.518 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:41:32.523 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:41:32.523 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:41:32.524 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:41:32.524 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_objectstore_tool td/ceph-helpers 2026-02-21T11:41:32.524 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: test_objectstore_tool: local dir=td/ceph-helpers 2026-02-21T11:41:32.524 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: test_objectstore_tool: setup td/ceph-helpers 2026-02-21T11:41:32.524 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:41:32.525 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:41:32.525 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:41:32.525 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:41:32.525 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:41:32.525 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:32.525 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:32.526 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:32.526 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:32.526 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:32.526 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:41:32.526 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:41:32.527 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:41:32.527 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:41:32.527 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:41:32.527 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:41:32.527 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:41:32.533 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:32.533 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:41:32.534 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:41:32.534 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:32.534 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:41:32.534 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:41:32.534 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:41:32.534 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:41:32.535 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:32.535 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:32.535 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:41:32.535 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:41:32.535 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:41:32.535 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:41:32.535 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:41:32.536 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:32.536 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:32.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:41:32.536 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:41:32.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:41:32.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:41:32.536 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:41:32.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:41:32.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:41:32.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:41:32.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:41:32.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:41:32.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:41:32.537 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:41:32.551 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:41:32.551 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:32.552 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:32.552 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:32.552 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:32.552 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:32.552 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:32.552 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:41:32.576 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:41:32.577 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:41:32.577 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:41:32.577 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:41:32.577 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:41:32.577 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:41:32.577 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:41:32.577 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:41:32.578 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:41:32.578 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:41:32.578 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:32.578 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:32.578 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:41:32.578 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:41:32.578 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:41:32.621 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:41:32.621 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:41:32.621 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:41:32.621 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:41:32.621 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:41:32.621 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:41:32.621 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:41:32.621 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:41:32.622 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:41:32.622 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:32.622 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:32.622 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:41:32.622 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:41:32.622 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:41:32.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: test_objectstore_tool: local osd=0 2026-02-21T11:41:32.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: test_objectstore_tool: run_mgr td/ceph-helpers x 2026-02-21T11:41:32.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:41:32.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:41:32.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:41:32.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:41:32.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:41:32.664 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:41:32.766 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:41:32.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:32.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:32.767 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:32.767 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:32.767 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:32.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:32.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:41:32.768 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:41:32.785 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: test_objectstore_tool: run_osd td/ceph-helpers 0 2026-02-21T11:41:32.785 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:41:32.785 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:41:32.785 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:41:32.785 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:41:32.786 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:41:32.786 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:41:32.786 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:41:32.786 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:41:32.786 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:41:32.786 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:41:32.787 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:41:32.787 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:41:32.787 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:41:32.787 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:41:32.787 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:41:32.787 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:32.787 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:32.787 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:32.787 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:32.788 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:32.788 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:32.788 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:32.788 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:41:32.788 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:41:32.788 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:41:32.788 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:41:32.788 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:41:32.788 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:41:32.788 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:41:32.788 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:41:32.788 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:41:32.789 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:41:32.789 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:41:32.789 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:41:32.789 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:41:32.789 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7e9e7639-e56f-4790-b8f5-aec2112f9617 2026-02-21T11:41:32.789 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 7e9e7639-e56f-4790-b8f5-aec2112f9617 2026-02-21T11:41:32.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7e9e7639-e56f-4790-b8f5-aec2112f9617' 2026-02-21T11:41:32.790 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:41:32.795 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDsmZlpxmpJLxAAx8jK/B8sge+C1Pc0YqNXxQ== 2026-02-21T11:41:32.795 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDsmZlpxmpJLxAAx8jK/B8sge+C1Pc0YqNXxQ=="}' 2026-02-21T11:41:32.796 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7e9e7639-e56f-4790-b8f5-aec2112f9617 -i td/ceph-helpers/0/new.json 2026-02-21T11:41:32.896 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:41:32.902 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:41:32.903 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsmZlpxmpJLxAAx8jK/B8sge+C1Pc0YqNXxQ== --osd-uuid 7e9e7639-e56f-4790-b8f5-aec2112f9617 2026-02-21T11:41:32.919 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:32.917+0000 7f89731fd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:32.920 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:32.919+0000 7f89731fd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:32.921 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:32.920+0000 7f89731fd780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:32.921 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:32.920+0000 7f89731fd780 -1 bdev(0x3180c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:41:32.921 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:32.920+0000 7f89731fd780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:41:33.403 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:41:33.403 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:41:33.403 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:41:33.403 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:41:33.404 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:41:33.509 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:41:33.510 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:41:33.511 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:41:33.511 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:41:33.511 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:41:33.512 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:41:33.523 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:33.521+0000 7f8502a5d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:33.524 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:33.523+0000 7f8502a5d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:33.525 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:33.524+0000 7f8502a5d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:33.606 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:41:33.606 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:41:33.606 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:41:33.606 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:41:33.606 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:41:33.607 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:41:33.607 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:41:33.608 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:41:33.608 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:41:33.608 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:41:33.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:41:33.725 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:33.723+0000 7f8502a5d780 -1 Falling back to public interface 2026-02-21T11:41:34.100 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:34.098+0000 7f8502a5d780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:41:34.704 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:41:34.704 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:41:34.704 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:41:34.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:41:34.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:41:34.705 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:41:34.802 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2583807872,v1:127.0.0.1:6801/2583807872] [v2:127.0.0.1:6802/2583807872,v1:127.0.0.1:6803/2583807872] exists,up 7e9e7639-e56f-4790-b8f5-aec2112f9617 2026-02-21T11:41:34.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:41:34.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:41:34.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:41:34.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: test_objectstore_tool: create_rbd_pool 2026-02-21T11:41:34.802 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:41:34.897 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:41:34.903 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:41:34.903 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:41:35.158 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:41:35.166 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:41:36.167 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:41:36.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: test_objectstore_tool: wait_for_clean 2026-02-21T11:41:36.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:41:36.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:41:36.446 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:41:36.446 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:41:36.446 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:36.446 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:36.446 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:36.446 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:36.446 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:36.530 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:41:36.531 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:41:36.531 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:41:36.531 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:41:36.531 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:41:36.531 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:41:36.753 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:41:36.753 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:41:36.753 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:41:36.753 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:41:36.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836483 2026-02-21T11:41:36.818 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836483 2026-02-21T11:41:36.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T11:41:36.819 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:41:36.819 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836483 2026-02-21T11:41:36.819 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:41:36.820 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:41:36.820 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836483 2026-02-21T11:41:36.820 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:41:36.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836483 2026-02-21T11:41:36.821 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T11:41:36.821 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836483 2026-02-21T11:41:36.822 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:37.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T11:41:37.049 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:41:38.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:41:38.051 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:38.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T11:41:38.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:41:39.280 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T11:41:39.280 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:39.517 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T11:41:39.517 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:41:39.518 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:41:39.518 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:41:39.829 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T11:41:39.829 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:41:39.829 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:41:39.829 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:41:39.829 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:41:39.829 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:41:39.829 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:41:39.829 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:41:40.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T11:41:40.053 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:41:40.053 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:41:40.053 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:41:40.370 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T11:41:40.370 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:41:40.370 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:41:40.371 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2026-02-21T11:41:40.391 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2026-02-21T11:41:40.391 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/ceph-helpers 2026-02-21T11:41:40.391 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-21T11:41:40.391 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-21T11:41:40.391 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-21T11:41:40.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2026-02-21T11:41:40.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/ceph-helpers 2026-02-21T11:41:40.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-21T11:41:40.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-21T11:41:40.392 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-21T11:41:40.393 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2026-02-21T11:41:40.393 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: diff - /etc/group 2026-02-21T11:41:40.393 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:40.393 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:40.393 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:40.393 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:40.394 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:40.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:41:40.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2026-02-21T11:41:40.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/ceph-helpers 2026-02-21T11:41:40.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-21T11:41:40.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-21T11:41:40.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-21T11:41:40.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2026-02-21T11:41:40.493 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:40.577 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:40.578 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:40.579 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:41:40.579 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:41:40.579 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:41:40.579 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:41:40.579 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:41:40.579 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:41:40.579 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:41:40.579 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-21T11:41:40.579 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-21T11:41:40.579 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:41:40.580 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-21T11:41:40.580 INFO:tasks.workunit.client.0.trial189.stderr:start osd.0 2026-02-21T11:41:40.580 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:41:40.580 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-21T11:41:40.581 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-21T11:41:40.581 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-21T11:41:40.581 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-21T11:41:40.581 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-21T11:41:40.592 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:40.590+0000 7f6772830780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:40.593 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:40.592+0000 7f6772830780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:40.594 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:40.592+0000 7f6772830780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:40.719 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:40.717+0000 7f6772830780 -1 Falling back to public interface 2026-02-21T11:41:40.807 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-21T11:41:40.808 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:41:40.808 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:41:40.808 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:41:40.808 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:41:40.808 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:41:40.808 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:41:40.808 INFO:tasks.workunit.client.0.trial189.stderr:0 2026-02-21T11:41:40.808 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:41:40.808 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:41:41.035 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:41:41.124 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:41.122+0000 7f6772830780 -1 osd.0 12 log_to_monitors true 2026-02-21T11:41:42.037 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:41:42.037 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:41:42.037 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:41:42.037 INFO:tasks.workunit.client.0.trial189.stderr:1 2026-02-21T11:41:42.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:41:42.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:41:42.264 INFO:tasks.workunit.client.0.trial189.stderr:osd.0 up in weight 1 up_from 15 up_thru 15 down_at 12 last_clean_interval [5,12) [v2:127.0.0.1:6800/367842301,v1:127.0.0.1:6801/367842301] [v2:127.0.0.1:6802/367842301,v1:127.0.0.1:6803/367842301] exists,up 7e9e7639-e56f-4790-b8f5-aec2112f9617 2026-02-21T11:41:42.264 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:41:42.264 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:41:42.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:41:42.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-21T11:41:42.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:41:42.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:41:42.265 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:41:42.265 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:41:42.265 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:42.265 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:42.265 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:42.265 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:42.265 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:42.346 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:41:42.346 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:41:42.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:41:42.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:41:42.347 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:41:42.347 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:41:42.569 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:41:42.569 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:41:42.569 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:41:42.569 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:41:42.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=64424509443 2026-02-21T11:41:42.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 64424509443 2026-02-21T11:41:42.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-64424509443' 2026-02-21T11:41:42.637 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:41:42.637 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-64424509443 2026-02-21T11:41:42.637 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:41:42.638 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:41:42.638 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-64424509443 2026-02-21T11:41:42.638 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:41:42.639 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=64424509443 2026-02-21T11:41:42.639 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 64424509443' 2026-02-21T11:41:42.639 INFO:tasks.workunit.client.0.trial189.stderr:waiting osd.0 seq 64424509443 2026-02-21T11:41:42.640 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:42.865 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 64424509443 -lt 64424509443 2026-02-21T11:41:42.865 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:41:42.866 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:41:42.866 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:41:43.182 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T11:41:43.182 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:41:43.182 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:41:43.182 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:41:43.183 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:41:43.183 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:41:43.183 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:41:43.183 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:41:43.408 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T11:41:43.408 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:41:43.409 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:41:43.409 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:41:43.723 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T11:41:43.723 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:41:43.723 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:41:43.723 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2026-02-21T11:41:43.723 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/ceph-helpers 2026-02-21T11:41:43.723 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-21T11:41:43.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-21T11:41:43.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-21T11:41:43.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2026-02-21T11:41:43.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/ceph-helpers 2026-02-21T11:41:43.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-21T11:41:43.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-21T11:41:43.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-21T11:41:43.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2026-02-21T11:41:43.724 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:43.724 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:43.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:43.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:43.725 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:43.827 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:41:43.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2026-02-21T11:41:43.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/ceph-helpers 2026-02-21T11:41:43.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-21T11:41:43.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-21T11:41:43.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-21T11:41:43.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2026-02-21T11:41:43.828 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2026-02-21T11:41:43.879 INFO:tasks.workunit.client.0.trial189.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2026-02-21T11:41:43.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2026-02-21T11:41:43.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: return 1 2026-02-21T11:41:43.914 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: return 1 2026-02-21T11:41:43.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: teardown td/ceph-helpers 2026-02-21T11:41:43.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:41:43.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:41:43.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:41:43.915 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:43.915 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:43.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:43.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:43.915 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:44.020 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:41:44.020 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:41:44.021 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:41:44.021 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:41:44.023 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:41:44.023 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:41:44.023 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:41:44.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:44.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:41:44.024 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:41:44.025 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:44.026 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:41:44.026 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:41:44.026 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:41:44.031 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:41:44.031 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:44.031 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:44.031 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:41:44.033 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:41:44.033 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:41:44.034 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:41:44.034 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_pg_schedule_scrub td/ceph-helpers 2026-02-21T11:41:44.034 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: test_pg_schedule_scrub: local dir=td/ceph-helpers 2026-02-21T11:41:44.034 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: test_pg_schedule_scrub: setup td/ceph-helpers 2026-02-21T11:41:44.034 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:41:44.034 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:41:44.034 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:41:44.034 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:41:44.034 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:41:44.034 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:41:44.034 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:41:44.035 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:41:44.035 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:41:44.035 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:41:44.036 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:41:44.036 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:41:44.036 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:41:44.036 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:41:44.037 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:41:44.037 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:41:44.037 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:41:44.037 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:44.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:41:44.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:41:44.038 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:41:44.039 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:41:44.039 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:41:44.039 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:41:44.040 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:41:44.040 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:44.040 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:44.041 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:41:44.041 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:41:44.041 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:41:44.041 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:41:44.043 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:41:44.043 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:44.043 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:44.043 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:41:44.044 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:41:44.044 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:41:44.044 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:41:44.044 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:41:44.045 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: test_pg_schedule_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:41:44.045 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:41:44.045 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:41:44.045 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:41:44.045 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:41:44.045 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:41:44.045 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:41:44.064 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:41:44.064 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:44.064 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:44.064 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:44.064 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:44.064 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:44.064 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:44.064 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:41:44.089 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:41:44.090 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:41:44.090 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:41:44.090 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:41:44.090 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:41:44.090 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:41:44.090 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:41:44.090 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:41:44.090 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:41:44.091 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:41:44.091 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:44.091 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:44.091 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:41:44.091 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:41:44.091 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:41:44.135 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:41:44.135 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:41:44.135 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:41:44.135 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:41:44.135 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:41:44.135 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:41:44.135 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:41:44.135 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:41:44.136 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:41:44.136 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:44.136 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:44.136 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:41:44.136 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:41:44.136 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:41:44.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: test_pg_schedule_scrub: run_mgr td/ceph-helpers x --mgr_stats_period=1 2026-02-21T11:41:44.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:41:44.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:41:44.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:41:44.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:41:44.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:41:44.179 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:41:44.278 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:41:44.278 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:44.278 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:44.278 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:44.278 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:44.278 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:44.278 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:44.278 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:41:44.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-21T11:41:44.298 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2021: test_pg_schedule_scrub: run_osd td/ceph-helpers 0 2026-02-21T11:41:44.298 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:41:44.298 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:41:44.298 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:41:44.299 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:41:44.299 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:41:44.299 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:41:44.299 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:41:44.299 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:41:44.299 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:41:44.299 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:41:44.299 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:41:44.299 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:41:44.299 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:41:44.299 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:41:44.300 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:41:44.300 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:41:44.300 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:41:44.300 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:41:44.300 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:41:44.300 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:41:44.300 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:44.300 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:41:44.300 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:41:44.300 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:41:44.300 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:41:44.300 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:41:44.300 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:41:44.301 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:41:44.301 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:41:44.301 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:41:44.301 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:41:44.301 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:41:44.301 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:41:44.301 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:41:44.301 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:41:44.301 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f2925cac-b958-4f0c-9782-b217a649238e 2026-02-21T11:41:44.301 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f2925cac-b958-4f0c-9782-b217a649238e' 2026-02-21T11:41:44.302 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 f2925cac-b958-4f0c-9782-b217a649238e 2026-02-21T11:41:44.302 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:41:44.308 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD4mZlpMrxKEhAAsQkMRwJo4QoIoDFqzxUQuQ== 2026-02-21T11:41:44.308 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD4mZlpMrxKEhAAsQkMRwJo4QoIoDFqzxUQuQ=="}' 2026-02-21T11:41:44.308 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f2925cac-b958-4f0c-9782-b217a649238e -i td/ceph-helpers/0/new.json 2026-02-21T11:41:44.410 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:41:44.416 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:41:44.417 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD4mZlpMrxKEhAAsQkMRwJo4QoIoDFqzxUQuQ== --osd-uuid f2925cac-b958-4f0c-9782-b217a649238e 2026-02-21T11:41:44.433 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:44.432+0000 7ff253468780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:44.435 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:44.433+0000 7ff253468780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:44.436 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:44.434+0000 7ff253468780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:44.436 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:44.435+0000 7ff253468780 -1 bdev(0x395cc00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:41:44.436 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:44.435+0000 7ff253468780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:41:44.857 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:41:44.858 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:41:44.858 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:41:44.858 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:41:44.859 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:41:44.961 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:41:44.962 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:41:44.962 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:41:44.962 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:41:44.962 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:41:44.962 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:41:44.976 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:44.975+0000 7f2d78e1d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:44.978 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:44.976+0000 7f2d78e1d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:44.979 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:44.977+0000 7f2d78e1d780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:41:45.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:41:45.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:41:45.060 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:41:45.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:41:45.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:41:45.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:41:45.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:41:45.061 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:41:45.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:41:45.061 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:41:45.158 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:41:45.193 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:45.192+0000 7f2d78e1d780 -1 Falling back to public interface 2026-02-21T11:41:45.593 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:41:45.591+0000 7f2d78e1d780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:41:46.159 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:41:46.159 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:41:46.160 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:41:46.160 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:41:46.160 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:41:46.160 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:41:46.255 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2253428509,v1:127.0.0.1:6801/2253428509] [v2:127.0.0.1:6802/2253428509,v1:127.0.0.1:6803/2253428509] exists,up f2925cac-b958-4f0c-9782-b217a649238e 2026-02-21T11:41:46.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:41:46.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:41:46.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:41:46.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: test_pg_schedule_scrub: create_rbd_pool 2026-02-21T11:41:46.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:41:46.347 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:41:46.353 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:41:46.353 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:41:46.664 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:41:46.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:41:47.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:41:47.952 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: test_pg_schedule_scrub: wait_for_clean 2026-02-21T11:41:47.952 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:41:47.952 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:41:47.952 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:41:47.953 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:41:47.954 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:47.954 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:47.954 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:47.954 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:47.954 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:48.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:41:48.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:41:48.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:41:48.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:41:48.038 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:41:48.038 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:41:48.262 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:41:48.263 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:41:48.263 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:41:48.263 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:41:48.331 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836484 2026-02-21T11:41:48.332 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836484 2026-02-21T11:41:48.332 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836484' 2026-02-21T11:41:48.332 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:41:48.332 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836484 2026-02-21T11:41:48.332 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:41:48.333 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:41:48.333 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836484 2026-02-21T11:41:48.333 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:41:48.334 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836484 2026-02-21T11:41:48.334 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836484 2026-02-21T11:41:48.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-21T11:41:48.335 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:48.561 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-21T11:41:48.561 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:41:49.562 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:41:49.563 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:49.789 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-21T11:41:49.789 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:41:50.790 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T11:41:50.791 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:51.018 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836486 -lt 21474836484 2026-02-21T11:41:51.019 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:41:51.019 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:41:51.019 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:41:51.331 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T11:41:51.332 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:41:51.332 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:41:51.332 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:41:51.332 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:41:51.332 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:41:51.332 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:41:51.332 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:41:51.556 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T11:41:51.557 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:41:51.557 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:41:51.557 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:41:51.871 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T11:41:51.871 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:41:51.871 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:41:51.872 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2024: test_pg_schedule_scrub: pg_schedule_scrub 1.0 2026-02-21T11:41:51.872 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1998: pg_schedule_scrub: local pgid=1.0 2026-02-21T11:41:51.872 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2000: pg_schedule_scrub: wait_for_pg_clean 1.0 2026-02-21T11:41:51.872 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-21T11:41:51.872 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-21T11:41:51.872 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:41:51.872 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:41:51.873 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:41:51.873 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:41:51.873 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:41:52.074 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-21T11:41:52.074 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-21T11:41:52.074 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-21T11:41:52.074 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-21T11:41:52.074 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:41:52.074 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:41:52.301 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:41:52.301 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:41:52.301 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:41:52.301 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:41:52.369 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836490 2026-02-21T11:41:52.369 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836490 2026-02-21T11:41:52.369 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836490' 2026-02-21T11:41:52.369 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:41:52.369 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836490 2026-02-21T11:41:52.370 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:41:52.371 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:41:52.371 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836490 2026-02-21T11:41:52.372 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:41:52.372 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836490 2026-02-21T11:41:52.372 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-21T11:41:52.372 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836490 2026-02-21T11:41:52.373 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:52.605 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836488 -lt 21474836490 2026-02-21T11:41:52.606 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:41:53.606 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:41:53.607 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:53.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836488 -lt 21474836490 2026-02-21T11:41:53.835 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:41:54.836 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T11:41:54.836 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:41:55.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836492 -lt 21474836490 2026-02-21T11:41:55.062 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T11:41:55.063 INFO:tasks.workunit.client.0.trial189.stdout:#---------- 1.0 loop 0 2026-02-21T11:41:55.064 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-21T11:41:55.064 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T11:41:55.065 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T11:41:55.065 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T11:41:55.065 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T11:41:55.065 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T11:41:55.129 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-21T11:41:55.130 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T11:41:55.130 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-21T11:41:55.130 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-21T11:41:55.130 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: pg_schedule_scrub: get_last_scrub_stamp 1.0 2026-02-21T11:41:55.130 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T11:41:55.130 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T11:41:55.130 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T11:41:55.130 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T11:41:55.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2001: pg_schedule_scrub: local last_scrub=2026-02-21T11:41:46.600458+0000 2026-02-21T11:41:55.354 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2002: pg_schedule_scrub: ceph tell 1.0 schedule-scrub 2026-02-21T11:41:55.417 INFO:tasks.workunit.client.0.trial189.stdout:{ 2026-02-21T11:41:55.417 INFO:tasks.workunit.client.0.trial189.stdout: "deep": false, 2026-02-21T11:41:55.418 INFO:tasks.workunit.client.0.trial189.stdout: "must": false, 2026-02-21T11:41:55.418 INFO:tasks.workunit.client.0.trial189.stdout: "stamp": "2026-02-18T23:41:55.416738+0000" 2026-02-21T11:41:55.418 INFO:tasks.workunit.client.0.trial189.stdout:} 2026-02-21T11:41:55.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2003: pg_schedule_scrub: wait_for_scrub 1.0 2026-02-21T11:41:46.600458+0000 2026-02-21T11:41:55.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2092: wait_for_scrub: local pgid=1.0 2026-02-21T11:41:55.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2093: wait_for_scrub: local last_scrub=2026-02-21T11:41:46.600458+0000 2026-02-21T11:41:55.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2094: wait_for_scrub: local sname=last_scrub_stamp 2026-02-21T11:41:55.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i=0 )) 2026-02-21T11:41:55.421 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i < 300 )) 2026-02-21T11:41:55.422 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T11:41:55.422 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T11:41:55.422 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T11:41:55.422 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T11:41:55.422 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T11:41:55.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: test 2026-02-21T11:41:46.600458+0000 '>' 2026-02-21T11:41:46.600458+0000 2026-02-21T11:41:55.644 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2100: wait_for_scrub: sleep 1 2026-02-21T11:41:56.645 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i++ )) 2026-02-21T11:41:56.645 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i < 300 )) 2026-02-21T11:41:56.646 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T11:41:56.646 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T11:41:56.647 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T11:41:56.647 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T11:41:56.647 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T11:41:56.869 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: test 2026-02-21T11:41:46.600458+0000 '>' 2026-02-21T11:41:46.600458+0000 2026-02-21T11:41:56.869 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2100: wait_for_scrub: sleep 1 2026-02-21T11:41:57.871 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i++ )) 2026-02-21T11:41:57.872 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i < 300 )) 2026-02-21T11:41:57.872 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T11:41:57.872 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T11:41:57.872 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T11:41:57.872 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T11:41:57.872 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T11:41:58.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: test 2026-02-21T11:41:46.600458+0000 '>' 2026-02-21T11:41:46.600458+0000 2026-02-21T11:41:58.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2100: wait_for_scrub: sleep 1 2026-02-21T11:41:59.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i++ )) 2026-02-21T11:41:59.095 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i < 300 )) 2026-02-21T11:41:59.096 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T11:41:59.096 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T11:41:59.096 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T11:41:59.096 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T11:41:59.096 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T11:41:59.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: test 2026-02-18T23:41:55.416738+0000 '>' 2026-02-21T11:41:46.600458+0000 2026-02-21T11:41:59.318 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2100: wait_for_scrub: sleep 1 2026-02-21T11:42:00.319 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i++ )) 2026-02-21T11:42:00.320 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i < 300 )) 2026-02-21T11:42:00.320 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T11:42:00.321 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T11:42:00.321 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T11:42:00.321 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T11:42:00.321 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T11:42:00.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: test 2026-02-21T11:41:57.775989+0000 '>' 2026-02-21T11:41:46.600458+0000 2026-02-21T11:42:00.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: wait_for_scrub: return 0 2026-02-21T11:42:00.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: test_pg_schedule_scrub: teardown td/ceph-helpers 2026-02-21T11:42:00.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:42:00.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:42:00.550 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:42:00.551 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:42:00.551 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:42:00.551 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:42:00.551 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:42:00.551 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:42:00.656 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:42:00.657 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:42:00.657 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:42:00.658 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:42:00.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:42:00.659 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:42:00.659 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:42:00.660 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:42:00.660 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:42:00.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:42:00.661 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:42:00.661 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:42:00.662 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:42:00.663 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:42:00.667 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:42:00.667 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:42:00.667 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:42:00.667 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:42:00.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:42:00.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:42:00.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: run_tests: for func in $funcs 2026-02-21T11:42:00.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2455: run_tests: test_pg_scrub td/ceph-helpers 2026-02-21T11:42:00.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: test_pg_scrub: local dir=td/ceph-helpers 2026-02-21T11:42:00.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: test_pg_scrub: setup td/ceph-helpers 2026-02-21T11:42:00.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-21T11:42:00.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-21T11:42:00.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T11:42:00.669 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-21T11:42:00.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T11:42:00.670 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:42:00.670 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:42:00.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:42:00.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:42:00.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:42:00.670 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:42:00.671 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T11:42:00.671 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T11:42:00.671 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T11:42:00.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T11:42:00.672 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T11:42:00.672 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T11:42:00.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:42:00.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T11:42:00.673 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T11:42:00.674 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T11:42:00.674 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T11:42:00.675 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-21T11:42:00.675 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T11:42:00.676 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T11:42:00.676 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:42:00.677 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:42:00.677 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T11:42:00.677 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T11:42:00.677 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T11:42:00.677 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-21T11:42:00.679 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-21T11:42:00.679 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:42:00.679 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:42:00.679 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.118434 2026-02-21T11:42:00.680 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-21T11:42:00.681 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-21T11:42:00.681 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-21T11:42:00.681 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-21T11:42:00.681 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:42:00.681 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-21T11:42:00.681 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-21T11:42:00.681 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-21T11:42:00.681 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-21T11:42:00.682 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-21T11:42:00.682 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:42:00.701 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-21T11:42:00.701 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:42:00.701 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:42:00.701 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:42:00.701 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:42:00.701 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:42:00.701 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:42:00.701 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-21T11:42:00.724 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-21T11:42:00.724 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-21T11:42:00.725 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:42:00.725 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:42:00.725 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-21T11:42:00.725 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-21T11:42:00.725 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:42:00.725 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:42:00.725 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:42:00.725 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:42:00.725 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:42:00.725 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:42:00.726 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:42:00.726 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:42:00.726 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get fsid 2026-02-21T11:42:00.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-21T11:42:00.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-21T11:42:00.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-21T11:42:00.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-21T11:42:00.767 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-21T11:42:00.768 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-21T11:42:00.768 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-21T11:42:00.768 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-21T11:42:00.768 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-21T11:42:00.768 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:42:00.768 INFO:tasks.workunit.client.0.trial189.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:42:00.768 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.118434/ceph-mon.a.asok 2026-02-21T11:42:00.768 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-21T11:42:00.768 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.118434/ceph-mon.a.asok config get mon_host 2026-02-21T11:42:00.811 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: test_pg_scrub: run_mgr td/ceph-helpers x 2026-02-21T11:42:00.811 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-21T11:42:00.811 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-21T11:42:00.811 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-21T11:42:00.811 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-21T11:42:00.811 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-21T11:42:00.812 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-21T11:42:00.912 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-21T11:42:00.912 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:42:00.912 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:42:00.912 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:42:00.912 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:42:00.912 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:42:00.912 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:42:00.913 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:42:00.913 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-21T11:42:00.931 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1975: test_pg_scrub: run_osd td/ceph-helpers 0 2026-02-21T11:42:00.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-21T11:42:00.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-21T11:42:00.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-21T11:42:00.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-21T11:42:00.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-21T11:42:00.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-21T11:42:00.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-21T11:42:00.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-21T11:42:00.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-21T11:42:00.932 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-21T11:42:00.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-21T11:42:00.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-21T11:42:00.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-21T11:42:00.933 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-21T11:42:00.933 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-21T11:42:00.933 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-21T11:42:00.933 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-21T11:42:00.933 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-21T11:42:00.933 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T11:42:00.933 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T11:42:00.934 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:42:00.934 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' 2026-02-21T11:42:00.934 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-21T11:42:00.934 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-21T11:42:00.934 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-21T11:42:00.934 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-21T11:42:00.934 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-21T11:42:00.934 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-21T11:42:00.934 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-21T11:42:00.934 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-21T11:42:00.934 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-21T11:42:00.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-21T11:42:00.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-21T11:42:00.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-21T11:42:00.935 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-21T11:42:00.935 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1840be7f-bd92-4695-b132-323e23852048 2026-02-21T11:42:00.935 INFO:tasks.workunit.client.0.trial189.stdout:add osd0 1840be7f-bd92-4695-b132-323e23852048 2026-02-21T11:42:00.936 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1840be7f-bd92-4695-b132-323e23852048' 2026-02-21T11:42:00.936 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-21T11:42:00.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAImplpvbcAOBAAH2okUU3efup9asM9FGeqfA== 2026-02-21T11:42:00.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAImplpvbcAOBAAH2okUU3efup9asM9FGeqfA=="}' 2026-02-21T11:42:00.941 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1840be7f-bd92-4695-b132-323e23852048 -i td/ceph-helpers/0/new.json 2026-02-21T11:42:01.045 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:42:01.052 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-21T11:42:01.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAImplpvbcAOBAAH2okUU3efup9asM9FGeqfA== --osd-uuid 1840be7f-bd92-4695-b132-323e23852048 2026-02-21T11:42:01.069 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:42:01.067+0000 7f4fe4012780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:42:01.070 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:42:01.069+0000 7f4fe4012780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:42:01.071 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:42:01.069+0000 7f4fe4012780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:42:01.071 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:42:01.070+0000 7f4fe4012780 -1 bdev(0x2e34c00 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-21T11:42:01.071 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:42:01.070+0000 7f4fe4012780 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-21T11:42:01.541 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-21T11:42:01.542 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-21T11:42:01.542 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-21T11:42:01.542 INFO:tasks.workunit.client.0.trial189.stdout:adding osd0 key to auth repository 2026-02-21T11:42:01.542 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-21T11:42:01.649 INFO:tasks.workunit.client.0.trial189.stdout:start osd.0 2026-02-21T11:42:01.649 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-21T11:42:01.649 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=76aaf8f9-4f5d-4917-8336-f5556d7f8649 --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.118434/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-21T11:42:01.649 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-21T11:42:01.649 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-21T11:42:01.649 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-21T11:42:01.663 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:42:01.661+0000 7fc3e7a14780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:42:01.665 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:42:01.663+0000 7fc3e7a14780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:42:01.665 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:42:01.664+0000 7fc3e7a14780 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-21T11:42:01.747 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-21T11:42:01.748 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-21T11:42:01.748 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-21T11:42:01.748 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-21T11:42:01.748 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-21T11:42:01.748 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:42:01.748 INFO:tasks.workunit.client.0.trial189.stdout:0 2026-02-21T11:42:01.749 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-21T11:42:01.749 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:42:01.749 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:42:01.843 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-21T11:42:01.851 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:42:01.850+0000 7fc3e7a14780 -1 Falling back to public interface 2026-02-21T11:42:02.190 INFO:tasks.workunit.client.0.trial189.stderr:2026-02-21T11:42:02.188+0000 7fc3e7a14780 -1 osd.0 0 log_to_monitors true 2026-02-21T11:42:02.844 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-21T11:42:02.844 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-21T11:42:02.844 INFO:tasks.workunit.client.0.trial189.stdout:1 2026-02-21T11:42:02.845 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-21T11:42:02.845 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-21T11:42:02.845 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-21T11:42:02.946 INFO:tasks.workunit.client.0.trial189.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2957192681,v1:127.0.0.1:6801/2957192681] [v2:127.0.0.1:6802/2957192681,v1:127.0.0.1:6803/2957192681] exists,up 1840be7f-bd92-4695-b132-323e23852048 2026-02-21T11:42:02.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-21T11:42:02.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-21T11:42:02.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-21T11:42:02.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1976: test_pg_scrub: create_rbd_pool 2026-02-21T11:42:02.947 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-21T11:42:03.041 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' does not exist 2026-02-21T11:42:03.047 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-21T11:42:03.047 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-21T11:42:03.326 INFO:tasks.workunit.client.0.trial189.stderr:pool 'rbd' already exists 2026-02-21T11:42:03.333 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-21T11:42:04.335 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-21T11:42:04.613 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1977: test_pg_scrub: wait_for_clean 2026-02-21T11:42:04.614 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-21T11:42:04.614 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-21T11:42:04.614 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-21T11:42:04.614 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-21T11:42:04.615 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:42:04.615 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:42:04.615 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:42:04.615 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:42:04.615 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:42:04.698 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-21T11:42:04.698 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-21T11:42:04.698 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-21T11:42:04.698 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-21T11:42:04.698 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:42:04.698 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:42:04.924 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:42:04.924 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:42:04.924 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:42:04.925 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:42:04.989 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836483 2026-02-21T11:42:04.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836483 2026-02-21T11:42:04.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836483' 2026-02-21T11:42:04.990 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:42:04.990 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836483 2026-02-21T11:42:04.990 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:42:04.991 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:42:04.991 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836483 2026-02-21T11:42:04.991 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:42:04.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836483 2026-02-21T11:42:04.993 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-21T11:42:04.993 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836483 2026-02-21T11:42:04.993 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:42:05.219 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T11:42:05.219 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:42:06.220 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:42:06.220 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:42:06.449 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-21T11:42:06.449 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:42:07.450 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T11:42:07.450 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:42:07.678 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-21T11:42:07.678 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-21T11:42:07.679 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:42:07.679 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:42:07.995 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-21T11:42:07.995 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-21T11:42:07.995 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-21T11:42:07.995 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-21T11:42:07.995 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-21T11:42:07.995 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-21T11:42:07.995 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-21T11:42:07.995 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-21T11:42:08.218 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-21T11:42:08.218 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-21T11:42:08.219 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-21T11:42:08.219 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-21T11:42:08.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-21T11:42:08.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-21T11:42:08.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-21T11:42:08.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: test_pg_scrub: pg_scrub 1.0 2026-02-21T11:42:08.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1952: pg_scrub: local pgid=1.0 2026-02-21T11:42:08.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: pg_scrub: wait_for_pg_clean 1.0 2026-02-21T11:42:08.549 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-21T11:42:08.549 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-21T11:42:08.550 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:42:08.550 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:42:08.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:42:08.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:42:08.550 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:42:08.752 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-21T11:42:08.752 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-21T11:42:08.753 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-21T11:42:08.753 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-21T11:42:08.753 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=300 2026-02-21T11:42:08.753 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:42:08.983 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:42:08.983 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:42:08.983 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:42:08.984 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-21T11:42:09.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq=21474836486 2026-02-21T11:42:09.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z 21474836486 2026-02-21T11:42:09.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: flush_pg_stats: seqs=' 0-21474836486' 2026-02-21T11:42:09.050 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: flush_pg_stats: for s in $seqs 2026-02-21T11:42:09.050 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: echo 0-21474836486 2026-02-21T11:42:09.050 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: cut -d - -f 1 2026-02-21T11:42:09.051 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: flush_pg_stats: osd=0 2026-02-21T11:42:09.052 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: echo 0-21474836486 2026-02-21T11:42:09.052 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: cut -d - -f 2 2026-02-21T11:42:09.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2296: flush_pg_stats: seq=21474836486 2026-02-21T11:42:09.053 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2297: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-21T11:42:09.053 INFO:tasks.workunit.client.0.trial189.stdout:waiting osd.0 seq 21474836486 2026-02-21T11:42:09.053 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:42:09.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836484 -lt 21474836486 2026-02-21T11:42:09.279 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:42:10.280 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-21T11:42:10.280 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:42:10.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836484 -lt 21474836486 2026-02-21T11:42:10.507 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2299: flush_pg_stats: sleep 1 2026-02-21T11:42:11.508 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2300: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-21T11:42:11.508 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-21T11:42:11.736 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2298: flush_pg_stats: test 21474836486 -lt 21474836486 2026-02-21T11:42:11.737 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T11:42:11.737 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-21T11:42:11.737 INFO:tasks.workunit.client.0.trial189.stdout:#---------- 1.0 loop 0 2026-02-21T11:42:11.738 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T11:42:11.738 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T11:42:11.738 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T11:42:11.738 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T11:42:11.738 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T11:42:11.803 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-21T11:42:11.803 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T11:42:11.803 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-21T11:42:11.804 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-21T11:42:11.804 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1955: pg_scrub: get_last_scrub_stamp 1.0 2026-02-21T11:42:11.804 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T11:42:11.804 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T11:42:11.804 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T11:42:11.804 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T11:42:12.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1955: pg_scrub: local last_scrub=2026-02-21T11:42:03.261654+0000 2026-02-21T11:42:12.028 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1956: pg_scrub: ceph pg scrub 1.0 2026-02-21T11:42:12.247 INFO:tasks.workunit.client.0.trial189.stderr:instructing pg 1.0 on osd.0 to scrub 2026-02-21T11:42:12.255 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1957: pg_scrub: wait_for_scrub 1.0 2026-02-21T11:42:03.261654+0000 2026-02-21T11:42:12.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2092: wait_for_scrub: local pgid=1.0 2026-02-21T11:42:12.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2093: wait_for_scrub: local last_scrub=2026-02-21T11:42:03.261654+0000 2026-02-21T11:42:12.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2094: wait_for_scrub: local sname=last_scrub_stamp 2026-02-21T11:42:12.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i=0 )) 2026-02-21T11:42:12.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i < 300 )) 2026-02-21T11:42:12.256 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T11:42:12.256 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T11:42:12.256 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T11:42:12.256 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T11:42:12.256 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T11:42:12.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: test 2026-02-21T11:42:03.261654+0000 '>' 2026-02-21T11:42:03.261654+0000 2026-02-21T11:42:12.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2100: wait_for_scrub: sleep 1 2026-02-21T11:42:13.477 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i++ )) 2026-02-21T11:42:13.478 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i < 300 )) 2026-02-21T11:42:13.478 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T11:42:13.478 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T11:42:13.478 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T11:42:13.479 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T11:42:13.479 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T11:42:13.701 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: test 2026-02-21T11:42:03.261654+0000 '>' 2026-02-21T11:42:03.261654+0000 2026-02-21T11:42:13.701 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2100: wait_for_scrub: sleep 1 2026-02-21T11:42:14.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i++ )) 2026-02-21T11:42:14.702 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i < 300 )) 2026-02-21T11:42:14.703 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T11:42:14.703 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T11:42:14.703 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T11:42:14.703 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T11:42:14.703 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T11:42:14.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: test 2026-02-21T11:42:03.261654+0000 '>' 2026-02-21T11:42:03.261654+0000 2026-02-21T11:42:14.925 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2100: wait_for_scrub: sleep 1 2026-02-21T11:42:15.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i++ )) 2026-02-21T11:42:15.926 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2096: wait_for_scrub: (( i < 300 )) 2026-02-21T11:42:15.926 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-21T11:42:15.926 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-21T11:42:15.926 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-21T11:42:15.927 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-21T11:42:15.927 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-21T11:42:16.149 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: wait_for_scrub: test 2026-02-21T11:42:12.193083+0000 '>' 2026-02-21T11:42:03.261654+0000 2026-02-21T11:42:16.149 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: wait_for_scrub: return 0 2026-02-21T11:42:16.149 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2026-02-21T11:42:16.149 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T11:42:16.149 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T11:42:16.150 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T11:42:16.150 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T11:42:16.150 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T11:42:16.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T11:42:16.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: test_pg_scrub: WAIT_FOR_CLEAN_TIMEOUT=10 2026-02-21T11:42:16.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: test_pg_scrub: TIMEOUT=2 2026-02-21T11:42:16.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: test_pg_scrub: pg_scrub 1.0 2026-02-21T11:42:16.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1952: pg_scrub: local pgid=1.0 2026-02-21T11:42:16.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1954: pg_scrub: wait_for_pg_clean 1.0 2026-02-21T11:42:16.256 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-21T11:42:16.256 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 10 1 3 2026-02-21T11:42:16.257 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-21T11:42:16.257 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-21T11:42:16.257 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-21T11:42:16.257 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-21T11:42:16.257 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-21T11:42:16.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '1') 2026-02-21T11:42:16.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-21T11:42:16.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-21T11:42:16.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-21T11:42:16.290 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: flush_pg_stats: local timeout=2 2026-02-21T11:42:16.290 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ceph osd ls 2026-02-21T11:42:16.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: flush_pg_stats: ids=0 2026-02-21T11:42:16.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: flush_pg_stats: seqs= 2026-02-21T11:42:16.519 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: flush_pg_stats: for osd in $ids 2026-02-21T11:42:16.520 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: timeout 2 ceph tell osd.0 flush_pg_stats 2026-02-21T11:42:18.524 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: seq= 2026-02-21T11:42:18.524 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: flush_pg_stats: true 2026-02-21T11:42:18.524 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: flush_pg_stats: test -z '' 2026-02-21T11:42:18.524 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: flush_pg_stats: continue 2026-02-21T11:42:18.524 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T11:42:18.524 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-21T11:42:18.524 INFO:tasks.workunit.client.0.trial189.stdout:#---------- 1.0 loop 0 2026-02-21T11:42:18.525 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T11:42:18.525 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T11:42:18.525 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T11:42:18.525 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T11:42:18.525 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T14:39:24.182 DEBUG:teuthology.orchestra.run:got remote process result: 124 2026-02-21T14:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-21T14:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1: is_pg_clean: teardown td/ceph-helpers 1 2026-02-21T14:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-21T14:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=1 2026-02-21T14:39:24.203 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-21T14:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-21T14:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-21T14:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-21T14:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-21T14:39:24.204 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-21T14:39:24.290 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-21T14:39:24.290 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-21T14:39:24.292 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-21T14:39:24.292 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-21T14:39:24.294 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-21T14:39:24.294 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-21T14:39:24.294 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-21T14:39:24.296 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T14:39:24.296 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-21T14:39:24.296 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-21T14:39:24.296 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-21T14:39:24.297 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-21T14:39:24.298 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 1 = 1 ']' 2026-02-21T14:39:24.298 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: '[' -n '' ']' 2026-02-21T14:39:24.298 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:185: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2026-02-21T14:39:24.299 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:186: teardown: mv td/ceph-helpers/mgr.x.log td/ceph-helpers/mon.a.log td/ceph-helpers/osd.0.log /home/ubuntu/cephtest/archive/log 2026-02-21T14:39:24.300 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-21T14:39:24.303 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-21T14:39:24.303 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-21T14:39:24.303 INFO:tasks.workunit.client.0.trial189.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.118434 2026-02-21T14:39:24.303 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.118434 2026-02-21T14:39:24.305 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-21T14:39:24.305 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-21T14:39:24.305 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-21T14:39:24.305 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 0 >= 5 )) 2026-02-21T14:39:24.305 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 1 2026-02-21T14:39:25.306 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-21T14:39:25.306 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-21T14:39:25.306 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 1' 2026-02-21T14:39:25.306 INFO:tasks.workunit.client.0.trial189.stdout:#---------- 1.0 loop 1 2026-02-21T14:39:25.307 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-21T14:39:25.307 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-21T14:39:25.307 INFO:tasks.workunit.client.0.trial189.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-21T14:39:25.307 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-21T14:39:25.307 INFO:tasks.workunit.client.0.trial189.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-21T14:40:24.203 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2026-02-21T14:41:24.204 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2026-02-21T14:41:24.204 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2026-02-21T14:41:24.205 DEBUG:teuthology.orchestra.run.trial189:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-21T14:41:24.543 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) ^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph-c_f7273f3049146bdd7fa7b4a400114872a50228e9/qa/tasks/workunit.py", line 144, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_f7273f3049146bdd7fa7b4a400114872a50228e9/qa/tasks/workunit.py", line 292, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_ceph_ceph-c_f7273f3049146bdd7fa7b4a400114872a50228e9/qa/tasks/workunit.py", line 433, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/test-ceph-helpers.sh) on trial189 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' 2026-02-21T14:41:24.549 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-21T14:41:24.566 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-21T14:41:24.566 DEBUG:teuthology.orchestra.run.trial189:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-21T14:41:24.692 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-21T14:41:24.703 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-21T14:41:24.703 DEBUG:teuthology.orchestra.run.trial189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-21T14:41:24.719 INFO:teuthology.orchestra.run.trial189.stderr:bash: line 1: ntpq: command not found 2026-02-21T14:41:24.745 INFO:teuthology.orchestra.run.trial189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2026-02-21T14:41:24.745 INFO:teuthology.orchestra.run.trial189.stdout:=============================================================================== 2026-02-21T14:41:24.745 INFO:teuthology.orchestra.run.trial189.stdout:^- ntp1.wiktel.com 1 8 377 246 -1145us[-1153us] +/- 20ms 2026-02-21T14:41:24.745 INFO:teuthology.orchestra.run.trial189.stdout:^- 144.202.66.214.vultruser> 3 10 377 77 +307us[ +307us] +/- 32ms 2026-02-21T14:41:24.745 INFO:teuthology.orchestra.run.trial189.stdout:^* pool-138-89-14-60.mad.ea> 1 9 237 128 -53us[ -61us] +/- 3099us 2026-02-21T14:41:24.745 INFO:teuthology.orchestra.run.trial189.stdout:^- 167-248-62-201.oa02.lnk0> 3 10 377 536 +731us[ +724us] +/- 73ms 2026-02-21T14:41:24.746 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-21T14:41:24.757 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-21T14:41:24.758 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-21T14:41:24.773 DEBUG:teuthology.orchestra.run.trial189:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2026-02-21T14:41:24.836 DEBUG:teuthology.orchestra.run.trial189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2026-02-21T14:41:24.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T14:41:24.862 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-21T14:41:24.886 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-21T14:41:24.900 INFO:teuthology.task.internal:Duration was 11680.489137 seconds 2026-02-21T14:41:24.901 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-21T14:41:24.915 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-21T14:41:24.915 DEBUG:teuthology.orchestra.run.trial189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-21T14:41:24.954 INFO:teuthology.orchestra.run.trial189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2026-02-21T14:41:25.211 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-21T14:41:25.211 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial189.front.sepia.ceph.com 2026-02-21T14:41:25.212 DEBUG:teuthology.orchestra.run.trial189:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-21T14:41:25.270 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-21T14:41:25.270 DEBUG:teuthology.orchestra.run.trial189:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-21T14:41:25.373 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-21T14:41:25.373 DEBUG:teuthology.orchestra.run.trial189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-21T14:41:25.396 INFO:teuthology.orchestra.run.trial189.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-21T14:41:25.397 INFO:teuthology.orchestra.run.trial189.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-21T14:41:25.397 INFO:teuthology.orchestra.run.trial189.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-21T14:41:25.397 INFO:teuthology.orchestra.run.trial189.stderr: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-21T14:41:25.397 INFO:teuthology.orchestra.run.trial189.stderr:/home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-21T14:41:25.408 INFO:teuthology.orchestra.run.trial189.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: 83.7% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-21T14:41:25.410 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-21T14:41:25.425 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-21T14:41:25.425 DEBUG:teuthology.orchestra.run.trial189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-21T14:41:25.474 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-21T14:41:25.490 DEBUG:teuthology.orchestra.run.trial189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-21T14:41:25.536 INFO:teuthology.orchestra.run.trial189.stdout:kernel.core_pattern = core 2026-02-21T14:41:25.549 DEBUG:teuthology.orchestra.run.trial189:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-21T14:41:25.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T14:41:25.605 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-21T14:41:25.621 INFO:teuthology.task.internal:Transferring archived files... 2026-02-21T14:41:25.624 DEBUG:teuthology.misc:Transferring archived files from trial189:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/skanta-2026-02-21_10:47:12-rados-wip-bharath21-testing-2026-02-20-1039-distro-default-trial/62535/remote/trial189 2026-02-21T14:41:25.625 DEBUG:teuthology.orchestra.run.trial189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-21T14:41:39.080 INFO:teuthology.task.internal:Removing archive directory... 2026-02-21T14:41:39.080 DEBUG:teuthology.orchestra.run.trial189:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-21T14:41:39.172 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-21T14:41:39.185 INFO:teuthology.task.internal:Not uploading archives. 2026-02-21T14:41:39.185 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-21T14:41:39.199 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-21T14:41:39.199 DEBUG:teuthology.orchestra.run.trial189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-21T14:41:39.214 INFO:teuthology.orchestra.run.trial189.stdout: 917576 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 21 14:41 /home/ubuntu/cephtest 2026-02-21T14:41:39.214 INFO:teuthology.orchestra.run.trial189.stdout: 1061675 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 21 11:30 /home/ubuntu/cephtest/mnt.0 2026-02-21T14:41:39.214 INFO:teuthology.orchestra.run.trial189.stdout: 1061676 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 21 11:39 /home/ubuntu/cephtest/mnt.0/client.0 2026-02-21T14:41:39.215 INFO:teuthology.orchestra.run.trial189.stdout: 1195506 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 21 11:39 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-21T14:41:39.215 INFO:teuthology.orchestra.run.trial189.stdout: 1195507 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 21 14:39 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2026-02-21T14:41:39.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-21T14:41:39.215 INFO:teuthology.orchestra.run.trial189.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2026-02-21T14:41:39.216 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_fa17720d0088c3ac28e473468bfc79eeaff5cd38/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial189 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2026-02-21T14:41:39.217 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-21T14:41:39.228 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-21T14:41:39.243 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on trial189 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2026-02-21T14:41:39.248 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/misc} duration: 11680.489137172699 failure_reason: 'Command failed (workunit test misc/test-ceph-helpers.sh) on trial189 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7273f3049146bdd7fa7b4a400114872a50228e9 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh''' flavor: default owner: scheduled_skanta@soko04.front.sepia.ceph.com sentry_event: null status: fail success: false 2026-02-21T14:41:39.248 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-21T14:41:39.382 INFO:teuthology.run:FAIL